In most cases, we start building our Django projects with a small number of Django apps. At some point, our projects grow and apps become larger which complicates the code maintenance. And we start thinking about refactoring - splitting a big Django app into smaller logically separated apps.
We create a new app and move the corresponding model(s) there.
Taking into consideration the fact that under the hood Django automatically derives the name of the database table from the name of your model class and the app that contains it, we decide to make migrations (moving model(s) into another app will force Django to look into different table name than it was before).
A model’s database table name is constructed by joining the model’sapp label– the name you used inmanage.py startapp– to the model’s class name, with an underscore between them(%app label%_%model name%).
If you have an called app cars (manage.py startapp cars), a model defined as classCarwill have a database table namedcars_car.
After taking a look into newly generated migration files, we see that instead of detecting a database table name change, Django detects a new model(table) in the new app that needs to be created and a model(table) that needs to be deleted in the old app. That means, if we apply newly generated migrations, we will lose the table and the data corresponding to the old-app-model and we will have a new and empty table which represent the new-app-model in database.
Often we are facing this problem when we have some important data in the moved model, even more, we have other models that are dependents(have foreign keys) of the moved model. In this case, auto-generated migrations won't help you to solve this problem yet(Django 3.0.5), unless you are really bored and looking for some crazy adventures 🌶.
In this article, we will discuss two migration strategies that will help us to move model(s) into other apps without deleting or creating new tables and/or foreign keys.
To demonstrate the strategy, we are going to create a simple project:
$ django-admin startproject mobilestorewith one django app:
mobilestore/$ django-admin startapp phonesand with a couple of models in it:
phones/models.py
from django.db import models
class Brand(models.Model):
name = models.CharField(max_length=128)
class Phone(models.Model):
model = models.CharField(max_length=128)
brand = models.ForeignKey(Brand, on_delete=models.CASCADE)
registering phones app:
mobilestore/settings.py
INSTALLED_APPS = [
...
'phones',
...
]
generating migrations:
mobilestore/$ python manage.py makemigrations phones
Migrations for 'phones':
phones/migrations/0001_initial.py
- Create model Brand
- Create model Phone
outputting SQL behind the new migration file:
mobilestore/$ python manage.py sqlmigrate phones 0001
BEGIN;
--
-- Create model Brand
--
CREATE TABLE "phones_brand" -- <<< name of the table
("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "name" varchar(128) NOT NULL);
--
-- Create model Phone
--
CREATE TABLE "phones_phone" -- <<< name of the table
("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "model" varchar(128) NOT NULL, "brand_id" integer NOT NULL REFERENCES "phones_brand" ("id") DEFERRABLE INITIALLY DEFERRED);
CREATE INDEX "phones_phone_brand_id_b4e25eb0" ON "phones_phone" ("brand_id");
COMMIT;
As we expected, Django generated names for new tables using the combination of app_label and model class name.
Now, let's apply them:
mobilestore/$ python manage.py migrate phones
Operations to perform:
Apply all migrations: phones
Running migrations:
Applying phones.0001_initial... OK
After a while, for decoupling purposes, we decide to create a new **brands** app and move the Brand model there:
mobilestore/$ django-admin startapp brandsbrands/models.py
from django.db import models
class Brand(models.Model):
name = models.CharField(max_length=128)
then we replace also to parameter value in Phone model with 'brands.Brand' lazy reference.
phones/models.py
from django.db import models
class Phone(models.Model):
model = models.CharField(max_length=128)
brand = models.ForeignKey('brands.Brand', on_delete=models.CASCADE)
registering brands app:
mobilestore/settings.py
INSTALLED_APPS = [
...
'phones',
'brands', # list the brands app
]
making migrations:
mobilestore/$ python manage.py makemigrations
Migrations for 'brands':
brands/migrations/0001_initial.py
- Create model Brand
Migrations for 'phones':
phones/migrations/0002_auto_20200418_1348.py
- Alter field brand on phone
- Delete model Brand <<< will delete the table
Now we have two new migration files - one in the brands app, the other in phones app. The new migration in the phones app depends on the one in the brands app, so let's check the migration file in the brands app first:
brands/migrations/0001_initial.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
]
operations = [
migrations.CreateModel(
name='Brand',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=128)),
],
),
]
corresponding SQL:
mobilestore/$ python manage.py sqlmigrate brands 0001
BEGIN;
--
-- Create model Brand
--
CREATE TABLE "brands_brand" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "name" varchar(128) NOT NULL);
COMMIT;
As you can see, Django is proposing to create a new table for the already moved Brand model instead of renaming the name of the table for it.
Now let's take a look at the other migration file:
phones/migrations/0002_auto_20200418_1348.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('brands', '0001_initial'),
('phones', '0001_initial'),
]
operations = [
migrations.AlterField(
model_name='phone',
name='brand',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='brands.Brand'),
),
migrations.DeleteModel(
name='Brand',
),
]
corresponding SQL:
mobilestore/$ python manage.py sqlmigrate phones 0002
BEGIN;
--
-- Alter field brand on phone
--
CREATE TABLE "new__phones_phone" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "brand_id" integer NOT NULL REFERENCES "brands_brand" ("id") DEFERRABLE INITIALLY DEFERRED, "model" varchar(128) NOT NULL);
INSERT INTO "new__phones_phone" ("id", "model", "brand_id") SELECT "id", "model", "brand_id" FROM "phones_phone";
DROP TABLE "phones_phone";
ALTER TABLE "new__phones_phone" RENAME TO "phones_phone";
CREATE INDEX "phones_phone_brand_id_b4e25eb0" ON "phones_phone" ("brand_id");
--
-- Delete model Brand
--
DROP TABLE "phones_brand";
COMMIT;
If you look at the first part of the SQL output, you'll find that Django is proposing to CREATE a new temporary new__phones_phone table with a foreign key to the brands__brand table, then INSERT data from existing phones_phone to the new__phones_phone table, after DROP phones_phone table and finally, RENAME the name of the table new__phones_phone to phones_phone(as it was before).
In the second part of the SQL we see DROP query for phones_brand table.
Well that's not what we want. How we can solve this problem? See in the next two sections. 👇
To solve this problem, we are going to modify the last 2 generated migration files that we saw above. We will use SeparateDatabaseAndState class to add operations that will reflect our changes to the model state, so we can avoid breaking Django's auto-detection system.
Let's start from brands app:
brands/migrations/0001_initial.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
]
# renamed operations
state_operations = [
migrations.CreateModel(
name='Brand',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=128)),
],
),
]
# defined new `operations` list
operations = [
migrations.SeparateDatabaseAndState(state_operations=state_operations),
]
All we did is renamed existing operations list variable to state_operations and defined a new operations list variable with the SeparateDatabaseAndState class object in it.
After this change, Django's auto detector won't be confused and it won't try to create a new table cause we don't have any database operation. Let's prove that:
mobilestore/$ python manage.py sqlmigrate brands 0001
BEGIN;
--
-- Custom state/database change combination
--
COMMIT;
Now, we are ready with the brands app, but things a bit complicated in the phones app.
With the help of AlterModelTable class we will rename the table name for model Brand, cause we moved it to a new app. We will define it in a new database_operations list variable.
phones/migrations/0002_auto_20200418_1348.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('brands', '0001_initial'),
('phones', '0001_initial'),
]
database_operations = [
migrations.AlterModelTable(
name='Brand',
table='brands_brand',
),
]
operations = [
migrations.AlterField(
model_name='phone',
name='brand',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='brands.Brand'),
),
migrations.DeleteModel(
name='Brand',
),
]
next change will be moving auto generated AlterField operation from operations to the database_operations list variable:
phones/migrations/0002_auto_20200418_1348.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('brands', '0001_initial'),
('phones', '0001_initial'),
]
database_operations = [
migrations.AlterModelTable(
name='Brand',
table='brands_brand',
),
migrations.AlterField(
model_name='phone',
name='brand',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='brands.Brand'),
),
]
operations = [
migrations.DeleteModel(
name='Brand',
),
]
The last thing that is left to do is to tell Django(state) that we are going to "delete" the Brand model that was in the phones app. We will just rename existing operations variable to state_operations and will define a final operations variable combining database and state operations from the Migration class:
phones/migrations/0002_auto_20200418_1348.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
atomic = False. # SQLite only
dependencies = [
('brands', '0001_initial'),
('phones', '0001_initial'),
]
database_operations = [
migrations.AlterModelTable(
name='Brand',
table='brands_brand',
),
migrations.AlterField(
model_name='phone',
name='brand',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='brands.Brand'),
),
]
state_operations = [
migrations.DeleteModel(
name='Brand',
),
]
operations = [
migrations.SeparateDatabaseAndState(
database_operations=database_operations,
state_operations=state_operations,
),
]
What it will do in terms of SQL:
mobilestore/$ python manage.py sqlmigrate phones 0002
ALTER TABLE "phones_brand" RENAME TO "brands_brand";
CREATE TABLE "new__phones_phone" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "model" varchar(128) NOT NULL, "brand_id" integer NOT NULL REFERENCES "brands_brand" ("id") DEFERRABLE INITIALLY DEFERRED);
INSERT INTO "new__phones_phone" ("id", "model", "brand_id") SELECT "id", "model", "brand_id" FROM "phones_phone";
DROP TABLE "phones_phone";
ALTER TABLE "new__phones_phone" RENAME TO "phones_phone";
CREATE INDEX "phones_phone_brand_id_b4e25eb0" ON "phones_phone" ("brand_id");
IMPORTANT! Everything is almost as we need. But the problem with this solution is that we will have some downtime during data transfer. As we saw, the SQL queries for creating a temporary table for Phone and moving data there will take some time, and this may cause some problems. To see how to avoid them let's just into the second solution.
Instead of renaming the name of the table for Brand model we will specify db_table model Meta option for it:
brands/models.py
from django.db import models
class Brand(models.Model):
name = models.CharField(max_length=128)
class Meta:
db_table = 'phones_brand'
By this change we are telling Django to continue using phones_brand as a table for Brand model.
Now all changes in auto-generated migrations must be defined as state_operations. After all changes, migrations files should look like below:
brands/migrations/0001_initial.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
]
# renamed operations
state_operations = [
migrations.CreateModel(
name='Brand',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=128)),
],
# some meta data
options={
'db_table': 'phones_brand',
},
),
]
operations = [
migrations.SeparateDatabaseAndState(state_operations=state_operations),
]
phones/migrations/0002_auto_20200418_1348.py
# Generated by Django 3.0.5 once upon a time :)
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('brands', '0001_initial'),
('phones', '0001_initial'),
]
# renamed operations
state_operations = [
migrations.AlterField(
model_name='phone',
name='brand',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='brands.Brand'),
),
migrations.DeleteModel(
name='Brand',
),
]
operations = [
migrations.SeparateDatabaseAndState(state_operations=state_operations),
]
Now we have no database operations and everything is as we want. ORM is also happy with that. We can apply the migrations without any doubts.
As we have noticed, the way Django generates a name for tables is not perfect. Specifying db_tables beforehand will be the best options in this case:
phones/models.py
from django.db import models
class Brand(models.Model):
name = models.CharField(max_length=128)
class Meta:
# name is independent of app_label
db_table = 'brands'
class Phone(models.Model):
model = models.CharField(max_length=128)
brand = models.ForeignKey(Brand, on_delete=models.CASCADE)
class Meta:
# name is independent of app_label
db_table = 'phones'