Allow custom formatting of objectID

[mkilmanas]

Description
Right now, SearchEntity has a fixed method to generate the objectID from doctrine's primary key. The problem is that objectIDs are limited in what characters they can contain (e.g. : is disallowed) - so this limits what values can be used in the Doctrine's entity fields if the entity has to be indexable/searchable.
We need a possibility to override/customize this method in order to map entity data to acceptable objectID.

Basic example
In our case we have CustomerAccount entity, which has a composite primary key made of two columns 'context' and 'account_id'. However, account_ids sometimes contain : character, which is not allowed in ObjectID, so these entities fail to to be indexed.

Other
There are multiple ways how this could be addressed, here's a few rough ideas:

• Introduce IdNormalizerInterface so that custom service can be implemented and then it would be used to generate the id for the SearchEntity
• Allow configuration to tell which entity getter should be called to get the objectID for it
• Generally make the current method cleanup the value to be compatible with the engine requirements (as long as ObjectID is never used for mapping back to the entity). Or it could use even use a hash function for objectID (and include the raw values as attributes)

[brunoocasali]

Hey @mkilmanas. I agree with implementing this!

We might not have enough time to work on it in the following weeks. Also, I will add some details to this issue to make the implementation easier. If someone wants to contribute and implement it, it will be released faster! PRs are always welcome!

Also, there is some related issues to this topic: #66 and #58

[mkilmanas]

I see Aggregator model (which is then extended by Aggregator Entity and Document) has similar logic (mapping entity to object ID and back) - maybe that the way to go, and make it equally apply to all Entities, not only those extending Aggregator (extending somehow doesn't sound right at this point in general)?

Although to me it still feels that an external service (like Normalizer/Denormalizer) would be the cleanest solution.

And from those other threads - did I get it right, there is a desire to completely get rid of objectID field? Or just how it is produced/used? Or just the naming?

[alifs-dev]

Formatting the document id
The document id must be an integer or a string. If the id is a string, it can only contain alphanumeric characters (a-z, A-Z, 0-9), hyphens (-), and underscores (_).

https://www.meilisearch.com/docs/learn/getting_started/primary_key#formatting-the-document-id

[alifs-dev]

sorry i don't know how to export diff ;)

config/packages/meilisearch.yaml

meilisearch:
  composite_entity:
    #  (a-z, A-Z, 0-9), hyphens (-), and underscores (_) // meilisearch doc
     id_separator: '-'
     field_separator: '_'

Changes for composite entities ids:

search-bundle/src/DependencyInjection/Configuration.php

->defaultValue('psr18.http_client')
                ->end()
...
                ->arrayNode('composite_entity')
                    ->addDefaultsIfNotSet()
                    ->children()
                        ->scalarNode('field_separator')
                            ->cannotBeEmpty()
                            ->defaultValue('__')
                        ->end()
                        ->scalarNode('id_separator')
                            ->isRequired()
                            ->cannotBeEmpty()
                            ->defaultValue('-')
                        ->end()
                    ->end()
                ->end()
...
                ->arrayNode('indices')

search-bundle/src/SearchableEntity.php

<?php

declare(strict_types=1);

namespace Meilisearch\Bundle;

use Doctrine\ORM\Mapping\ClassMetadata;
use Symfony\Component\Config\Definition\Exception\Exception;
use Symfony\Component\HttpKernel\Kernel;
use Symfony\Component\Serializer\Exception\ExceptionInterface;
use Symfony\Component\Serializer\Normalizer\DateTimeNormalizer;
use Symfony\Component\Serializer\Normalizer\NormalizableInterface;
use Symfony\Component\Serializer\Normalizer\NormalizerInterface;
use Meilisearch\Bundle\Collection;

final class SearchableEntity
{
    private string $indexUid;

    /** @var object */
    private $entity;

    /** @var ClassMetadata<object> */
    private ClassMetadata $entityMetadata;

    private ?NormalizerInterface $normalizer;

    /**
     * @var list<string>
     */
    private array $normalizationGroups;

    /** @var int|string */
    private $id;

    private $configuration;
    /**
     * @param object                $entity
     * @param ClassMetadata<object> $entityMetadata
     */
    public function __construct(
        string $indexUid,
        $entity,
        ClassMetadata $entityMetadata,
        ?NormalizerInterface $normalizer = null,
        array $extra = [],
        Collection $configuration
    ) {
        $this->indexUid = $indexUid;
        $this->entity = $entity;
        $this->entityMetadata = $entityMetadata;
        $this->normalizer = $normalizer;
        $this->normalizationGroups = $extra['normalizationGroups'] ?? [];
        $this->configuration = $configuration;
        $this->setId();
    }

    public function getIndexUid(): string
    {
        return $this->indexUid;
    }

    /**
     * @throws ExceptionInterface
     */
    public function getSearchableArray(): array
    {
        $context = [
            'meilisearch' => true,
            'fieldsMapping' => $this->entityMetadata->fieldMappings,
        ];

        if (\count($this->normalizationGroups) > 0) {
            $context['groups'] = $this->normalizationGroups;
        }

        if (Kernel::VERSION_ID >= 70100) {
            $context[DateTimeNormalizer::FORMAT_KEY] = 'U';
            $context[DateTimeNormalizer::CAST_KEY] = 'int';
        }

        if ($this->entity instanceof NormalizableInterface && null !== $this->normalizer) {
            return $this->entity->normalize($this->normalizer, Searchable::NORMALIZATION_FORMAT, $context);
        }

        if (null !== $this->normalizer) {
            return $this->normalizer->normalize($this->entity, Searchable::NORMALIZATION_FORMAT, $context);
        }

        return [];
    }

    private function setId(): void
    {
        $ids = $this->entityMetadata->getIdentifierValues($this->entity);

        if (0 === \count($ids)) {
            throw new Exception('Entity has no primary key');
        }

        if (! $this->entityMetadata->isIdentifierComposite) {
            $this->id = reset($ids);
        } else {
            $objectID = [];

            $compositeEntityConfig = $this->configuration->get('composite_entity');
            $field_separator = $compositeEntityConfig['field_separator'];
            $id_separator = $compositeEntityConfig['id_separator'];

            foreach ($ids as $key => $value) {
                $objectID[] = $key.$id_separator.$value;
            }

            $this->id = join($field_separator, $objectID);
        }
    }

    /**
     * @return int|string
     */
    public function getId()
    {
        return $this->id;
    }
}

search-bundle/src/Services/MeilisearchService.php

<?php

declare(strict_types=1);

namespace Meilisearch\Bundle\Services;

use Doctrine\Common\Util\ClassUtils;
use Doctrine\ORM\Mapping\LegacyReflectionFields;
use Doctrine\ORM\Proxy\DefaultProxyClassNameResolver;
use Doctrine\Persistence\ObjectManager;
use Meilisearch\Bundle\Collection;
use Meilisearch\Bundle\Engine;
use Meilisearch\Bundle\Entity\Aggregator;
use Meilisearch\Bundle\Exception\ObjectIdNotFoundException;
use Meilisearch\Bundle\Exception\SearchHitsNotFoundException;
use Meilisearch\Bundle\SearchableEntity;
use Meilisearch\Bundle\SearchService;
use Symfony\Component\Config\Definition\Exception\Exception;
use Symfony\Component\PropertyAccess\PropertyAccess;
use Symfony\Component\PropertyAccess\PropertyAccessorInterface;
use Symfony\Component\Serializer\Normalizer\NormalizerInterface;

final class MeilisearchService implements SearchService
{
    private NormalizerInterface $normalizer;
    private Engine $engine;
    private Collection $configuration;
    private PropertyAccessorInterface $propertyAccessor;
    /**
     * @var list<class-string>
     */
    private array $searchableEntities;
    /**
     * @var array<class-string, list<class-string>>
     */
    private array $entitiesAggregators;
    /**
     * @var list<class-string<Aggregator>>
     */
    private array $aggregators;
    /**
     * @var array<class-string, array<string>>
     */
    private array $classToSerializerGroup;
    private array $indexIfMapping;

    public function __construct(NormalizerInterface $normalizer, Engine $engine, array $configuration, ?PropertyAccessorInterface $propertyAccessor = null)
    {
        $this->normalizer = $normalizer;
        $this->engine = $engine;
        $this->configuration = new Collection($configuration);
        $this->propertyAccessor = $propertyAccessor ?? PropertyAccess::createPropertyAccessor();

        $this->setSearchableEntities();
        $this->setAggregatorsAndEntitiesAggregators();
        $this->setClassToSerializerGroup();
        $this->setIndexIfMapping();
    }

    public function isSearchable($className): bool
    {
        $className = $this->getBaseClassName($className);

        return \in_array($className, $this->searchableEntities, true);
    }

    public function getSearchable(): array
    {
        return $this->searchableEntities;
    }

    public function getConfiguration(): Collection
    {
        return $this->configuration;
    }

    public function searchableAs(string $className): string
    {
        $className = $this->getBaseClassName($className);

        $indexes = new Collection($this->getConfiguration()->get('indices'));
        $index = $indexes->firstWhere('class', $className);

        return $this->getConfiguration()->get('prefix').$index['name'];
    }

    public function index(ObjectManager $objectManager, $searchable): array
    {
        $searchable = \is_array($searchable) ? $searchable : [$searchable];
        $searchable = array_merge($searchable, $this->getAggregatorsFromEntities($objectManager, $searchable));

        $dataToIndex = array_filter(
            $searchable,
            fn ($entity) => $this->isSearchable($entity)
        );

        $dataToRemove = [];
        foreach ($dataToIndex as $key => $entity) {
            if (!$this->shouldBeIndexed($entity)) {
                unset($dataToIndex[$key]);
                $dataToRemove[] = $entity;
            }
        }

        if (\count($dataToRemove) > 0) {
            $this->remove($objectManager, $dataToRemove);
        }

        return $this->makeSearchServiceResponseFrom(
            $objectManager,
            $dataToIndex,
            fn ($chunk) => $this->engine->index($chunk)
        );
    }

    public function remove(ObjectManager $objectManager, $searchable): array
    {
        $searchable = \is_array($searchable) ? $searchable : [$searchable];
        $searchable = array_merge($searchable, $this->getAggregatorsFromEntities($objectManager, $searchable));

        $searchable = array_filter(
            $searchable,
            fn ($entity) => $this->isSearchable($entity)
        );

        return $this->makeSearchServiceResponseFrom(
            $objectManager,
            $searchable,
            fn ($chunk) => $this->engine->remove($chunk)
        );
    }

    public function clear(string $className): array
    {
        $this->assertIsSearchable($className);

        return $this->engine->clear($this->searchableAs($className));
    }

    public function deleteByIndexName(string $indexName): ?array
    {
        return $this->engine->delete($indexName);
    }

    public function delete(string $className): ?array
    {
        $this->assertIsSearchable($className);

        return $this->engine->delete($this->searchableAs($className));
    }

    public function search(
        ObjectManager $objectManager,
        string $className,
        string $query = '',
        array $searchParams = []
    ): array {
        $this->assertIsSearchable($className);

        $ids = $this->engine->search($query, $this->searchableAs($className), $searchParams + ['limit' => $this->configuration['nbResults']]);
        $results = [];

        // Check if the engine returns results in "hits" key
        if (!isset($ids[self::RESULT_KEY_HITS])) {
            throw new SearchHitsNotFoundException(\sprintf('There is no "%s" key in the search results.', self::RESULT_KEY_HITS));
        }

        foreach ($ids[self::RESULT_KEY_HITS] as $hit) {
            if (!isset($hit[self::RESULT_KEY_OBJECTID])) {
                throw new ObjectIdNotFoundException(\sprintf('There is no "%s" key in the result.', self::RESULT_KEY_OBJECTID));
            }

            $documentId = $hit[self::RESULT_KEY_OBJECTID];
            $entityClass = $className;

            if (\in_array($className, $this->aggregators, true)) {
                $objectId = $hit[self::RESULT_KEY_OBJECTID];
                $entityClass = $className::getEntityClassFromObjectId($objectId);
                $documentId = $className::getEntityIdFromObjectId($objectId);
            }

            $factory = $objectManager->getMetadataFactory();

            if ($factory->getMetadataFor(ltrim($className, '\\'))->isIdentifierComposite) {
                $config = $this->getConfiguration();
                $compositeEntityConfig = $config->get('composite_entity');
                $field_separator = $compositeEntityConfig['field_separator'];
                $id_separator = $compositeEntityConfig['id_separator'];

                $identifiers = explode($field_separator, $documentId);
                $documentId = [];
                foreach($identifiers as $field){
                    list($key, $value) = explode($id_separator, $field);
                    $documentId[$key] = $value;
                }
            }

            $repo = $objectManager->getRepository($entityClass);
            $entity = $repo->find($documentId);

            if (null !== $entity) {
                $results[] = $entity;
            }
        }

        return $results;
    }

    public function rawSearch(
        string $className,
        string $query = '',
        array $searchParams = []
    ): array {
        $this->assertIsSearchable($className);

        return $this->engine->search($query, $this->searchableAs($className), $searchParams);
    }

    public function count(string $className, string $query = '', array $searchParams = []): int
    {
        $this->assertIsSearchable($className);

        return $this->engine->count($query, $this->searchableAs($className), $searchParams);
    }

    public function shouldBeIndexed(object $entity): bool
    {
        $className = $this->getBaseClassName($entity);

        $propertyPath = $this->indexIfMapping[$className];

        if (null !== $propertyPath) {
            if ($this->propertyAccessor->isReadable($entity, $propertyPath)) {
                return (bool) $this->propertyAccessor->getValue($entity, $propertyPath);
            }

            return false;
        }

        return true;
    }

    /**
     * @param object|class-string $objectOrClass
     *
     * @return class-string
     */
    private function getBaseClassName($objectOrClass): string
    {
        foreach ($this->searchableEntities as $class) {
            if (is_a($objectOrClass, $class, true)) {
                return $class;
            }
        }

        if (\is_object($objectOrClass)) {
            return self::resolveClass($objectOrClass);
        }

        return $objectOrClass;
    }

    private function setSearchableEntities(): void
    {
        $searchable = [];
        foreach ($this->configuration->get('indices') as $index) {
            $searchable[] = $index['class'];
        }
        $this->searchableEntities = array_unique($searchable);
    }

    private function setAggregatorsAndEntitiesAggregators(): void
    {
        $this->entitiesAggregators = [];
        $this->aggregators = [];

        foreach ($this->configuration->get('indices') as $index) {
            if (is_subclass_of($index['class'], Aggregator::class)) {
                foreach ($index['class']::getEntities() as $entityClass) {
                    if (!isset($this->entitiesAggregators[$entityClass])) {
                        $this->entitiesAggregators[$entityClass] = [];
                    }

                    $this->entitiesAggregators[$entityClass][] = $index['class'];
                    $this->aggregators[] = $index['class'];
                }
            }
        }

        $this->aggregators = array_unique($this->aggregators);
    }

    private function setClassToSerializerGroup(): void
    {
        $mapping = [];

        /** @var array $indexDetails */
        foreach ($this->configuration->get('indices') as $indexDetails) {
            $mapping[$indexDetails['class']] = $indexDetails['enable_serializer_groups'] ? $indexDetails['serializer_groups'] : [];
        }
        $this->classToSerializerGroup = $mapping;
    }

    private function setIndexIfMapping(): void
    {
        $mapping = [];

        /** @var array $indexDetails */
        foreach ($this->configuration->get('indices') as $indexDetails) {
            $mapping[$indexDetails['class']] = $indexDetails['index_if'];
        }
        $this->indexIfMapping = $mapping;
    }

    /**
     * Returns the aggregators instances of the provided entities.
     *
     * @param array<int, object> $entities
     *
     * @return array<int, object>
     */
    private function getAggregatorsFromEntities(ObjectManager $objectManager, array $entities): array
    {
        $aggregators = [];

        foreach ($entities as $entity) {
            $entityClassName = self::resolveClass($entity);
            if (\array_key_exists($entityClassName, $this->entitiesAggregators)) {
                foreach ($this->entitiesAggregators[$entityClassName] as $aggregator) {
                    $aggregators[] = new $aggregator(
                        $entity,
                        $objectManager->getClassMetadata($entityClassName)->getIdentifierValues($entity)
                    );
                }
            }
        }

        return $aggregators;
    }

    /**
     * For each chunk performs the provided operation.
     *
     * @param array<int, object> $entities
     */
    private function makeSearchServiceResponseFrom(
        ObjectManager $objectManager,
        array $entities,
        callable $operation
    ): array {
        $batch = [];
        foreach (array_chunk($entities, $this->configuration->get('batchSize')) as $chunk) {
            $searchableEntitiesChunk = [];
            foreach ($chunk as $entity) {
                $entityClassName = $this->getBaseClassName($entity);

                $searchableEntitiesChunk[] = new SearchableEntity(
                    $this->searchableAs($entityClassName),
                    $entity,
                    $objectManager->getClassMetadata($entityClassName),
                    $this->normalizer,
                    ['normalizationGroups' => $this->getNormalizationGroups($entityClassName)],
                    $this->getConfiguration()
                );
            }

            $batch[] = $operation($searchableEntitiesChunk);
        }

        return $batch;
    }

    /**
     * @param class-string $className
     *
     * @return list<string>
     */
    private function getNormalizationGroups(string $className): array
    {
        return $this->classToSerializerGroup[$className];
    }

    private function assertIsSearchable(string $className): void
    {
        if (!$this->isSearchable($className)) {
            throw new Exception('Class '.$className.' is not searchable.');
        }
    }

    private static function resolveClass(object $object): string
    {
        static $resolver;

        $resolver ??= (function () {
            // Native lazy objects compatibility
            if (PHP_VERSION_ID >= 80400 && class_exists(LegacyReflectionFields::class)) {
                return fn (object $object) => \get_class($object);
            }

            // Doctrine ORM v3+ compatibility
            if (class_exists(DefaultProxyClassNameResolver::class)) {
                return fn (object $object) => DefaultProxyClassNameResolver::getClass($object);
            }

            // Legacy Doctrine ORM compatibility
            return fn (object $object) => ClassUtils::getClass($object); // @codeCoverageIgnore
        })();

        return $resolver($object);
    }
}

[norkunas]

Hey @mkilmanas,

If you still need this, could you check if #410 would work for you?

[norkunas]

@alifs-dev you also please check #410, I've added smart handling of composite ids