Building an Ever-Changing Mapping Map

The following task came up at work.

Certain JSON data arrives from an external source. It had to be reshaped to fit a required Map structure. When content that belonged in a single object arrived spread across several objects, it had to be merged into one. Conversely, information that should be separate had to be split apart. Sometimes keys or values had to be changed too.

The problem was that the structure of the incoming data could change from project to project. At first I designed an all-purpose mapSchema that could cover every case, but that was not a good approach. Since the input data changes with the project, the transformation logic also had to be flexibly modifiable. It had to be possible to specify and modify how data is transformed for each situation without changing code.

I abstracted the way data is processed, broke it down into atomic pieces, and designed a data processing pipeline. First I unified the input interface. The JSON data arriving from outside is flattened across the board, regardless of nesting depth. I simplified it so that the results are gathered into a List<Map<String,String>> for processing.

The revised mapSchema class can hold several custom Mappers. The actual data processing happens as the data passes through the accumulated Mappers one by one. That made the method very simple: run the Mappers in order to process the input data. The class that holds the Mappers was designed, for example, like this.

@Entity
public class MapSchema{
    @OneToMany
    private List<Mapper> mappers;
    
    public List<Map<String,String>> doMap (List<Map<String,String>> maps){
        mappers.foreach(Mapper::map);
        return maps;
    }
}

A Mapper is a class with two functions. First, it analyzes the Map it receives and decides whether this Mapper should process that data. If it is not data to process, it lets it pass through unchanged and analyzes the next data in order. This is the target-check function. The second function manipulates the data.

So I designed the structure of Mapper as follows.

@Entity
public class Mapper {

    @ManyToOne
    private MapSchema mapSchema;

    @OneToMany(mappedBy = "mapper")
    @OrderColumn
    private List<MapCondition> conditions = new ArrayList<>();

    @OneToMany(mappedBy = "mapper")
    @OrderColumn
    private List<MapFunction> functions = new ArrayList<>();

    public List<Map<String,String>> map (List<Map<String,String>> maps){

        List<Map<String,String>> targets = new ArrayList<>(maps);

        for(MapCondition condition : conditions){
            targets = targets.stream().filter(condition::test).toList();
        }

        maps.removeAll(targets);

        for(MapFunction function : functions){
            targets = function.apply(targets);
        }
        maps.addAll(targets);

        return maps;
    }
}

MapCondition and MapFunction are abstract classes.

@Inheritance(strategy = InheritanceType.JOINED)
@DiscriminatorColumn(name = "DTYPE")
@Entity
public abstract class MapCondition {
    
    @ManyToOne
    protected Mapper mapper;
    
    public abstract boolean test(Map<String,String> map);
}

@Inheritance(strategy = InheritanceType.JOINED)
@DiscriminatorColumn(name = "DTYPE")
@Entity
public abstract class MapFunction {
    
    @ManyToOne
    protected Mapper mapper;

    public abstract List<Map<String,String>> apply(List<Map<String,String>> maps);
}

Examples of classes that extend each abstract class are as follows.

@DiscriminatorValue(value = "KEY_VALUES_CONDITION")
@Entity(name = "map_condition_key_values_condition")
public class KeyValuesCondition extends MapCondition {

    private String targetKey;

    @Convert(converter = ListToStringConverter.class)
    private List<String> expectedValues;

    @Override
    public boolean test(Map<String, String> map) {
        String actualValue = map.get(targetKey);
        boolean isContained = expectedValues.contains(actualValue);
        return isContained;
    }
}

@DiscriminatorValue(value = "KEY_MAPPER")
@Entity(name = "map_function_key_mapper")
public class KeyMapper extends MapFunction {

    @Convert(converter = MapToStringConverter.class)
    private Map<String,String> keyMap = new HashMap<>();
    
    @Override
    public List<Map<String, String>> apply(List<Map<String, String>> maps) {
        for(Map<String, String> map : maps){
            keyMap.forEach((k, v) -> map.put(v, map.get(k)));
        }
        return maps;
    }
}

These examples omit the id field and the like.

With this design, several conditions can be put in one list and the targets to process identified through a multi-condition AND check. Several MapFunctions can also be put in a list to perform the desired manipulations over multiple passes. Since each is held in a list as an abstract class, you can create as many condition-check classes and manipulation classes as you need and make use of polymorphism.

I had some concerns while working out this design. I was worried about using @Entity too much. My first plan was to implement polymorphism by having @Embeddable classes implement a particular interface, but that would have meant implementing everything, data access included, directly at the application level, which was too much of a burden.

So I considered having them implement Serializable and storing the mapping objects in the database as serialized bytes to use whenever needed. But that seemed to offer little benefit; it would only raise complexity, and predicting how the methods would behave at runtime, and maintaining them, looked difficult.

So in the end I decided to break data manipulation down, as far as possible, into the smallest pieces that never need to change, and to design each as an entity extending a base class. I separated condition checks from manipulation functions, and designed the manipulation functions too, whether changing a key, changing a value, merging, or splitting, so that none depends on or contains another, and so that even complex mappings are possible just by running them in sequence. See below.

classDiagram
   class MapSchema {
       +Project project
       +Long pk
       +ImportType importType
       +List~Mapper~ mappers
       +insertNewMapper(Integer, Mapper) void
       +mapAll(List) List
   }
   class Mapper {
       +MapSchema mapSchema
       +Long pk
       +List~MapFunction~ functions
       +List~MapCondition~ conditions
       +map(List) List
   }
   class MapFunction {
       +Mapper mapper
       +Long pk
       +apply(List) List
   }
   class MapCondition {
       +Mapper mapper
       +Long pk
       +test(Map) boolean
   }
   class GroupByKeysMerger {
       +mergeGroupClassifier
       +groupByKeys
       +mergeMapFunction
   }
   class StringJoiner {
       +splitter
       +newKey
       +targetKey1
       +targetKey2
   }
   class KeyBasedSplitter {
       +splitKeys
   }
   class ValueMapper {
       +targetKey
       +valueMap
       +defaultValue
   }
   class KeyMapper {
       +keyMap
   }
   class KeyValuesCondition {
       +targetKey
       +expectedValues
   }

   MapSchema "1" *-- "*" Mapper : mappers
   Mapper "1" *-- "*" MapFunction : functions
   Mapper "1" *-- "*" MapCondition : conditions
   MapFunction <|-- GroupByKeysMerger
   MapFunction <|-- StringJoiner
   MapFunction <|-- KeyBasedSplitter
   MapFunction <|-- ValueMapper
   MapFunction <|-- KeyMapper
   MapCondition <|-- KeyValuesCondition

Next I started thinking about adding or modifying each of the conditions and functions through an API from outside. Using @JsonTypeInfo and @JsonSubTypes, I changed it so that the request DTO states a type and a different object is produced accordingly.

// request body
{
  "type": "KEY_VALUES_CONDITION",
  "targetKey": "name",
  "expectedValues": [
    "John Doe",
    "Jane Doe"
  ]
}

//dto
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, include = JsonTypeInfo.As.PROPERTY, property = "type")
@JsonSubTypes({
        @JsonSubTypes.Type(name = "KEY_VALUES_CONDITION", value = KeyValuesConditionDto.class)
})
public interface MapConditionDto {
    public MapCondition convertToNewOne(Mapper mapper);
}

public class KeyValuesConditionDto implements MapConditionDto {
    private Long mapConditionPk;
    private Long mapperPk;
    private String targetKey;
    private List<String> expectedValues = new ArrayList<>();
}

To reduce complexity, I did not create separate repositories or services for Mapper and the conditions and functions it is made of. Instead, I gave the MapSchema class, which actually assembles and uses them, the responsibility of managing its child entities as a kind of aggregate. With orphanRemoval, they are deleted automatically when their reference in the list is dropped.

public class MapSchema {
    
    @Builder.Default
    @OneToMany(mappedBy = "mapSchema", fetch = FetchType.LAZY, cascade = CascadeType.ALL, orphanRemoval = true)
    @OrderColumn
    private List<Mapper> mappers = new ArrayList<>();
    
    public void insertNewMapper(int index, Mapper mapper) {
        if(index < 0){
            mappers.addFirst(mapper);
            return;
        }
        if(index >= mappers.size()){
            mappers.add(mapper);
            return;
        }
        mappers.add(index, mapper);
    }
}

The advantage of this approach is that Mapper and its conditions and functions, which had to be made entities for the sake of polymorphism through inheritance, can be handled as if they were embeddable classes. But they are still entities, so identifiers still exist and are recorded. It is true that this leaves room for identifying and handling those entities by their primary keys.

So it is necessary to force Mapper and its component classes to be handled through a single path: the parent entity, MapSchema. The design should identify by MapSchema and address a Mapper by its index in the list. When designing the API, you can avoid exposing identifiers for objects that are not meant to be handled as entities, or make the entity's primary key private and leave out its getter. On top of that, modifying these objects can be designed as deleting the existing one and adding a new one, as if handling a kind of immutable object.