0.9.10

New contains/icontains Query Backend

This release introduces a new method of gathering results from icontains and contains queries. The previous implementation suffered the following drawbacks:

  • If a field was indexed for a contains-style query, the possible length of value of the field was limited. Saving a value which was too long would result in an error.
  • The index data for such fields could become very large, this would make entities "heavy" and would result in performance issues across the site as this index data would be transferred with the returned instances.
  • Index data was saved across several fields (due to a misunderstanding of the limits of list properties).

The new backend has none of these drawbacks. The limit for string value is the same as CharField (1500 bytes) and index data is stored on a descendent entity, meaning data isn't needlessly transferred with instances.

Migrating

If you have been using contains/icontains fields on versions of Djangae before 0.9.10 you will need to migrate your data. One potential method for this is:

  1. Examine djangaeidx.yaml and make a note of the models which have contains or icontains indexes.
  2. Deploy a version of your application to a non-default version (do not migrate traffic!). Make sure this version uses Djangae 0.9.10 and has DJANGAE_USE_LEGACY_CONTAINS_LOGIC set to False (default).
  3. Open a shell using the remote sandbox (./manage.py --sandbox=remote shell).
  4. Import each model in turn and use the defer_iteration to resave your instances on the new non-default version: e.g. defer_iteration(MyModel.objects.all(), MyModel.save, _target="your-new-app-version").
  5. Note that what you are doing here is using the remote shell to defer tasks onto the task queue of your production application, not to localhost.
  6. Test the version works as expected before migrating traffic to it.

Be aware that although this will add new index data, the original index data will still linger which will affect performance. You can use Djangae's migration support to write a migration file to clear this data. An example migration is:

from djangae.db.backends.appengine.indexing import (
    LegacyContainsIndexer,
    LegacyIContainsIndexer,
    CHARACTERS_PER_COLUMN,
    MAX_COLUMNS_PER_SPECIAL_INDEX,
)
from djangae.db.migrations.operations import RemoveFieldData
from djangae.fields import CharField
from django.db import migrations


class Migration(migrations.Migration):
    """ Populates the Person model with the default value for the new `favourite_colour` field. """

    dependencies = [
        # Add dependent migration here (at least `('app_label', '0001_initial')`)
    ]

    # Set this to the model name of your model
    MODEL_NAME = "YourModel"

    # Set this to the name (or column name, if different) of the indexed field on your model
    FIELD_COLUMN = "your_field_column_here"

    # The legacy `contains` indexing supported indexing of up to 103 characters per column, but if
    # you know that your data doesn't contain any values this long then you can reduce this, which
    # may reduce the number of operations needed to delete your old index data
    MAX_LENGTH_OF_INDEXED_VALUES = 103

    # Change this to LegacyIContainsIndexer if appropriate
    INDEXER = LegacyContainsIndexer()

    # END OF THINGS THAT YOU NEED TO EDIT

    # The legacy indexer(s) use multiple columns to split the indexed values across, so we need to
    # create some example values to get the correct column names for deletion
    values_for_lengths = [
        "a" * x for i, x in enumerate(CHARACTERS_PER_COLUMN)
        if i < MAX_COLUMNS_PER_SPECIAL_INDEX
    ]

    index_column_names = [
        INDEXER.indexed_column_name(FIELD_COLUMN, value, None)
        for value in values_for_lengths
    ]

    operations = [
        RemoveFieldData(
            MODEL_NAME,
            index_column_name,
            CharField()
        )
        for index_column_name in index_column_names
    ]

You will need one of these migrations for each contain or icontains entry in djangaeidx.yaml. Make sure you at least have an initial migration for the app (which can be generated using the makemigrations command).

Remember: You should not run a migration like this until you have resaved all your contains-using models and are successfully running with DJANGAE_USE_LEGACY_CONTAINS_LOGIC = False

For more info on migrations, see the migrations documentation.