How to safely move a model to another app in Django
Database migration strategies that help to move model(s) into other apps without deleting or creating new tables and/or foreign keys.
Intro
In most cases, we start building our Django projects with a small number of Django apps. At some point, our projects grow and apps become larger which complicates the code maintenance. And we start thinking about refactoring - splitting a big Django app into smaller logically separated apps.
We create a new app and move the corresponding model(s) there.
Taking into consideration the fact that under the hood Django automatically derives the name of the database table from the name of your model class and the app that contains it, we decide to make migrations (moving model(s) into another app will force Django to look into different table name than it was before).
A model’s database table name is constructed by joining the model’sapp label
– the name you used inmanage.py startapp
– to the model’s class name, with an underscore between them(%app label%_%model name%
).
If you have an called app cars (manage.py startapp cars
), a model defined as classCar
will have a database table namedcars_car
.
After taking a look into newly generated migration files, we see that instead of detecting a database table name change, Django detects a new model(table) in the new app that needs to be created and a model(table) that needs to be deleted in the old app. That means, if we apply newly generated migrations, we will lose the table and the data corresponding to the old-app-model and we will have a new and empty table which represent the new-app-model in database.
Often we are facing this problem when we have some important data in the moved model, even more, we have other models that are dependents(have foreign keys) of the moved model. In this case, auto-generated migrations won't help you to solve this problem yet(Django 3.0.5), unless you are really bored and looking for some crazy adventures 🌶.
In this article, we will discuss two migration strategies that will help us to move model(s) into other apps without deleting or creating new tables and/or foreign keys.
Getting started
To demonstrate the strategy, we are going to create a simple project:
$ django-admin startproject mobilestore
with one django app:
mobilestore/$ django-admin startapp phones
and with a couple of models in it:
phones/models.py
from django.db import models
class Brand(models.Model):
name = models.CharField(max_length=128)
class Phone(models.Model):
model = models.CharField(max_length=128)
brand = models.ForeignKey(Brand, on_delete=models.CASCADE)
registering phones
app:
mobilestore/settings.py
INSTALLED_APPS = [
...
'phones',
...
]
generating migrations:
mobilestore/$ python manage.py makemigrations phones
Migrations for 'phones':
phones/migrations/0001_initial.py
- Create model Brand
- Create model Phone
outputting SQL behind the new migration file:
mobilestore/$ python manage.py sqlmigrate phones 0001
BEGIN;
--
-- Create model Brand
--
CREATE TABLE "phones_brand" -- <<< name of the table
("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "name" varchar(128) NOT NULL);
--
-- Create model Phone
--
CREATE TABLE "phones_phone" -- <<< name of the table
("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "model" varchar(128) NOT NULL, "brand_id" integer NOT NULL REFERENCES "phones_brand" ("id") DEFERRABLE INITIALLY DEFERRED);
CREATE INDEX "phones_phone_brand_id_b4e25eb0" ON "phones_phone" ("brand_id");
COMMIT;
As we expected, Django generated names for new tables using the combination of app_label and model class name.
Now, let's apply them:
mobilestore/$ python manage.py migrate phones
Operations to perform:
Apply all migrations: phones
Running migrations:
Applying phones.0001_initial... OK
Problem
After a while, for decoupling purposes, we decide to create a new brands app and move the Brand
model there:
mobilestore/$ django-admin startapp brands
brands/models.py
from django.db import models
class Brand(models.Model):
name = models.CharField(max_length=128)
then we replace also to
parameter value in Phone
model with 'brands.Brand'
lazy reference.
phones/models.py
from django.db import models
class Phone(models.Model):
model = models.CharField(max_length=128)
brand = models.ForeignKey('brands.Brand', on_delete=models.CASCADE)
registering brands
app:
mobilestore/settings.py
INSTALLED_APPS = [
...
'phones',
'brands', # list the brands app
]
making migrations:
mobilestore/$ python manage.py makemigrations
Migrations for 'brands':
brands/migrations/0001_initial.py
- Create model Brand
Migrations for 'phones':
phones/migrations/0002_auto_20200418_1348.py
- Alter field brand on phone
- Delete model Brand <<< will delete the table
Now we have two new migration files - one in the brands
app, the other in phones
app. The new migration in the phones
app depends on the one in the brands
app, so let's check the migration file in the brands
app first:
brands/migrations/0001_initial.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
]
operations = [
migrations.CreateModel(
name='Brand',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=128)),
],
),
]
corresponding SQL:
mobilestore/$ python manage.py sqlmigrate brands 0001
BEGIN;
--
-- Create model Brand
--
CREATE TABLE "brands_brand" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "name" varchar(128) NOT NULL);
COMMIT;
As you can see, Django is proposing to create a new table for the already moved Brand
model instead of renaming the name of the table for it.
Now let's take a look at the other migration file:
phones/migrations/0002_auto_20200418_1348.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('brands', '0001_initial'),
('phones', '0001_initial'),
]
operations = [
migrations.AlterField(
model_name='phone',
name='brand',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='brands.Brand'),
),
migrations.DeleteModel(
name='Brand',
),
]
corresponding SQL:
mobilestore/$ python manage.py sqlmigrate phones 0002
BEGIN;
--
-- Alter field brand on phone
--
CREATE TABLE "new__phones_phone" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "brand_id" integer NOT NULL REFERENCES "brands_brand" ("id") DEFERRABLE INITIALLY DEFERRED, "model" varchar(128) NOT NULL);
INSERT INTO "new__phones_phone" ("id", "model", "brand_id") SELECT "id", "model", "brand_id" FROM "phones_phone";
DROP TABLE "phones_phone";
ALTER TABLE "new__phones_phone" RENAME TO "phones_phone";
CREATE INDEX "phones_phone_brand_id_b4e25eb0" ON "phones_phone" ("brand_id");
--
-- Delete model Brand
--
DROP TABLE "phones_brand";
COMMIT;
If you look at the first part of the SQL output, you'll find that Django is proposing to CREATE
a new temporary newphones_phone
table with a foreign key to the brandsbrand
table, then INSERT
data from existing phones_phone
to the newphones_phone
table, after DROP
phones_phone
table and finally, RENAME
the name of the table newphones_phone
to phones_phone
(as it was before).
In the second part of the SQL we see DROP
query for phones_brand
table.
Well that's not what we want. How we can solve this problem? See in the next two sections. 👇
Solution #1
To solve this problem, we are going to modify the last 2 generated migration files that we saw above. We will use SeparateDatabaseAndState class to add operations that will reflect our changes to the model state, so we can avoid breaking Django's auto-detection system.
Let's start from brands
app:
brands/migrations/0001_initial.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
]
# renamed operations
state_operations = [
migrations.CreateModel(
name='Brand',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=128)),
],
),
]
# defined new `operations` list
operations = [
migrations.SeparateDatabaseAndState(state_operations=state_operations),
]
All we did is renamed existing operations
list variable to state_operations
and defined a new operations
list variable with the SeparateDatabaseAndState
class object in it.
After this change, Django's auto detector won't be confused and it won't try to create a new table cause we don't have any database operation. Let's prove that:
mobilestore/$ python manage.py sqlmigrate brands 0001
BEGIN;
--
-- Custom state/database change combination
--
COMMIT;
Now, we are ready with the brands
app, but things a bit complicated in the phones
app.
With the help of AlterModelTable class we will rename the table name for model Brand
, cause we moved it to a new app. We will define it in a new database_operations
list variable.
phones/migrations/0002_auto_20200418_1348.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('brands', '0001_initial'),
('phones', '0001_initial'),
]
database_operations = [
migrations.AlterModelTable(
name='Brand',
table='brands_brand',
),
]
operations = [
migrations.AlterField(
model_name='phone',
name='brand',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='brands.Brand'),
),
migrations.DeleteModel(
name='Brand',
),
]
next change will be moving auto generated AlterField
operation from operations
to the database_operations
list variable:
phones/migrations/0002_auto_20200418_1348.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('brands', '0001_initial'),
('phones', '0001_initial'),
]
database_operations = [
migrations.AlterModelTable(
name='Brand',
table='brands_brand',
),
migrations.AlterField(
model_name='phone',
name='brand',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='brands.Brand'),
),
]
operations = [
migrations.DeleteModel(
name='Brand',
),
]
The last thing that is left to do is to tell Django(state) that we are going to "delete" the Brand
model that was in the phones
app. We will just rename existing operations
variable to state_operations
and will define a final operations
variable combining database and state operations from the Migration class:
phones/migrations/0002_auto_20200418_1348.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
atomic = False. # SQLite only
dependencies = [
('brands', '0001_initial'),
('phones', '0001_initial'),
]
database_operations = [
migrations.AlterModelTable(
name='Brand',
table='brands_brand',
),
migrations.AlterField(
model_name='phone',
name='brand',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='brands.Brand'),
),
]
state_operations = [
migrations.DeleteModel(
name='Brand',
),
]
operations = [
migrations.SeparateDatabaseAndState(
database_operations=database_operations,
state_operations=state_operations,
),
]
What it will do in terms of SQL:
mobilestore/$ python manage.py sqlmigrate phones 0002
ALTER TABLE "phones_brand" RENAME TO "brands_brand";
CREATE TABLE "new__phones_phone" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "model" varchar(128) NOT NULL, "brand_id" integer NOT NULL REFERENCES "brands_brand" ("id") DEFERRABLE INITIALLY DEFERRED);
INSERT INTO "new__phones_phone" ("id", "model", "brand_id") SELECT "id", "model", "brand_id" FROM "phones_phone";
DROP TABLE "phones_phone";
ALTER TABLE "new__phones_phone" RENAME TO "phones_phone";
CREATE INDEX "phones_phone_brand_id_b4e25eb0" ON "phones_phone" ("brand_id");
IMPORTANT! Everything is almost as we need. But the problem with this solution is that we will have some downtime during data transfer. As we saw, the SQL queries for creating a temporary table for Phone
and moving data there will take some time, and this may cause some problems. To see how to avoid them let's just into the second solution.
Solution #2
Instead of renaming the name of the table for Brand
model we will specify db_table model Meta option for it:
brands/models.py
from django.db import models
class Brand(models.Model):
name = models.CharField(max_length=128)
class Meta:
db_table = 'phones_brand'
By this change we are telling Django to continue using phones_brand
as a table for Brand
model.
Now all changes in auto-generated migrations must be defined as state_operations. After all changes, migrations files should look like below:
brands/migrations/0001_initial.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
]
# renamed operations
state_operations = [
migrations.CreateModel(
name='Brand',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=128)),
],
# some meta data
options={
'db_table': 'phones_brand',
},
),
]
operations = [
migrations.SeparateDatabaseAndState(state_operations=state_operations),
]
phones/migrations/0002_auto_20200418_1348.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('brands', '0001_initial'),
('phones', '0001_initial'),
]
# renamed operations
state_operations = [
migrations.AlterField(
model_name='phone',
name='brand',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='brands.Brand'),
),
migrations.DeleteModel(
name='Brand',
),
]
operations = [
migrations.SeparateDatabaseAndState(state_operations=state_operations),
]
Now we have no database operations and everything is as we want. ORM is also happy with that. We can apply the migrations without any doubts.
Conclusion
As we have noticed, the way Django generates a name for tables is not perfect. Specifying db_table
s beforehand will be the best options in this case:
phones/models.py
from django.db import models
class Brand(models.Model):
name = models.CharField(max_length=128)
class Meta:
# name is independent of app_label
db_table = 'brands'
class Phone(models.Model):
model = models.CharField(max_length=128)
brand = models.ForeignKey(Brand, on_delete=models.CASCADE)
class Meta:
# name is independent of app_label
db_table = 'phones'