dimension 과 autogen

'Match mode'를 'Auto generate'로 세팅하고 사용하는 dimension들이 그 value가 update 될 때 forge의 옵션에 따라 영향을 받는 듯 하다.
이과 관련해 알게된 몇 가지 forge 옵션을 소개한다.

1. Forge.args[2] = --ignoreState
 - 이 옵션은 말 그대로 state를 무시해라라는 옵션인 것 같다.
endeca는 인덱싱 때 state에 이전 인덱싱 때 autogen으로 생성되는 dimension의 status를 저장해두고 다음 인덱싱 때 이 state를 참고해 변경이 있는 dimension들만 업데이트를 하는 것 같다.
하지만 이 옵션이 주어지면 저 state를 무시하고 인덱싱이 되기 때문에 모든 autogen dimension의 status를 새로 업데이트를 하고 새로운 state를 만드는 듯 하다.

 - endeca definition
 "Instructs Forge to ignore any state files on startup. The state files are ignored only during the startup process. After start up, Forge creates state files during an update and overwrites the existing state files."

#. 이 옵션과 관련한 QA

Cleaning up your auto-generated dimensions files

Problem:
You have a large auto-generated dimensions file that is comprised of many dimensions and values that are either no longer valid for your project or you have normal dimensions that make them redundant and you want to clean up the file. However, you are concerned about losing your dimvals (dimension values) that may be used in various parts of your application when the dimval IDs are regenerated.

Cause:
When an autogen dimension is deleted or a dimval no longer exists in the data set, we do not automatically purge it from the autogen dimensions state files. It will maintain its location in the hierarchy and retain its dimval ID.

Solution:
You can run Forge with the --ignoreState flag. This will re-generate all dimension value ID's for autogen and external dimensions that Forge finds in your data and in your external taxonomy.

However, there are two major drawbacks to this procedure. The primary drawback is that any merchant rules and/or precedence rules that use these dimension values will be invalidated as the dimval ID's they are all built on will have changed. If you are referencing these dimval ID's in your application's code, these IDs will need to be remapped as well.

The second drawback is that loading these dimensions in Developer Studio will cause these dimension value IDs to be written to the external_dimensions.xml file, thus effectively locking them in place. This loading will load everything that is currently in your autogen state, which includes the dimension values you are trying to clean up.

2. Forge.args[2] = --noAutoGen
 - 이 옵션은 autogen을 사용하지 않겠다 (?)정도로 이해 하면 될 것 같다. 저 옵션이 주어진 후에 모든 autogen dimension들은 state에 있는 dimension data만 계속 보여지게 될 것이다. 즉, dimension은 저 옵션이 주어지기 직전의 인덱싱 때의 값을 계속 유지 할 것이다.

 - endeca definition
 "Do not generate new dimension value IDs (for incremental updates when batch processing is running)."

#. 사용 사례

Dimension server

The dimension server uses the two dimension adapters as sources.

The URL field (General tab) specifies the location to which the autogen_dimensions.xml.gz file is written. This file contains persistent dimension data produced by auto-generation.

There are some special considerations when using AutoGen classification with partial updates. When new dimension values are generated in the partial updates pipeline, the dimension changes are included in the updates sent to the MDEX Engine.

Because the baseline and partial update pipelines share the same autogen file, changes to AutoGen dimensions are also shared between the two. However, at any given time, only one of the two update processes can modify the Autogen_dimensions file.

Rather than suspend partial updates during baseline updates, Forge supports the --noAutoGen command-line option, which turns off the creation of new dimension values. Classification with existing dimension values continues normally, but classification failures result in no matching dimension values, rather than in the creation of new ones.

3. Forge.args[2] = --pruneAutoGen
 - 이 옵션은 dimension 중에 promote를 했을 때 각각의 value는 정적인 dimension id들을 할 당 받게 된다. 즉, 이 id와 중복되는 녀석이 이 후에 인덱싱 때 존재한다면 절대 안되다는 얘기다.
 "경험을 얘기하자면 'width'라는 dimension을 생성하고 이 녀석을 'promote'후에 'shift' 형태로 값을 할당하고 사용한 적이 있다. 그러다 이 'width'가 필요없게 되어 삭제하고 인덱싱을 했는데 dimension id가 중복이라고 에러가 나는 것이다. 그래서 그 id를 찾아봤지만 어디에도 없었다. -_-;
나중에야 알게 되었지만 그 id에 해당하는 값을 갖는 다른 dimension의 'promote' value가 있었던 것이다." 요럴 때 이 옵션을 사용하면 이전에 'promote'로 할당 된 값들을 날릴 수가 있다는거~~

#. endeca forum에 올라온 이 옵션을 사용한 방법

1) Load and promote the dimension. Promotion should convert the matching type of the dimension to Normal
2) Add --pruneAutoGen to the Forge command line options within Developer Studio -> Tools -> Manager Settings. This will clean out the old dimensions.
3) Push the config, run a baseline, retrieve the config.
4) Convert the pipeline's matching type of the dimension back to Auto Gen
5) Remove --pruneAutoGen again.
6) Push the config, run a baseline, retrieve the config.
7) Loading the dimension values should now reveal any values that are not internal.

#. 이 옵션과 관련한 글

Editing auto-generated dimension values

Normally, auto-generated dimension values cannot be edited. They are generated by Forge behind the scenes and maintained in state files. With an auto-generated dimension, you can configure the dimension's behavior, but you cannot configure the behavior of individual dimension values within the dimension. Endeca's load and promote functionality, however, allows you to load an auto-generated dimension and then promote its dimension values so that they become editable.

Loading versus promoting

The process of converting an auto-generated dimension has been broken down into two distinct steps, loading and promoting. Loading displays the auto-generated dimension values so that you can inspect them before promoting them. In addition, loaded dimension values can be used in the following ways. You can:

• Use them in dynamic business rules.

• Use them in precedence rules.

• Modify their ranking.

• Enable/disable their Inert and Collapsible features.

After loading a dimension, you have the option of promoting its dimension values. Promoting a dimension's values converts them to manual dimension values, with all of the editing capability of a regular manually created dimension value. Promotion is done on a per dimension basis. In other words, when you promote a dimension, all of its dimension values are promoted; you cannot pick individual dimension values to promote and leave others to be auto-generated. It is important to note that, after promotion, you can no longer treat a promoted dimension as auto-generated. All configuration and editing must be performed manually at this point.

Loading and promoting requirements

Loading and promoting auto-generated dimensions has two requirements:

• You must use Web Studio and EAC. Web Studio stores temporary copies of auto-generated dimensions. This is the location that Developer Studio retrieves them from during loading.

• A dimension must be auto-generated before it can be loaded and/or promoted. This means that you must have already run Forge at least once before attempting to load and promote auto-generated dimensions.

To load and promote an auto-generated dimension:

1. In Dimensions view, select an auto-generated dimension. Auto-generated dimensions are indicated by this icon:

2. Click Values to display the Dimension Values view. You see the root of the auto-generated dimension.

3. Click Load. The dimension is populated with the auto-generated dimension values.

4. Click Promote.

5. Click Yes to confirm the promotion.

6. After the promotion is complete, click OK. The icons next to the promoted dimension values change to indicate that they are now treated as manual dimensions:

Cleaning up after promoting dimensions

After you promote auto-generated dimension values, you must run a baseline update with Forge’s --pruneAutoGen flag. The flag cleans out any promoted dimensions from the auto-generated state files. This step is necessary in order to avoid any potential duplicate dimensions in your output records.

To clean up after promoting auto-generated dimensions:

1. Make sure the --pruneAutoGen flag is specified for Forge. (This flag is specified by default for new projects).

2. Send the latest instance configuration to Web Studio.

3. Run a baseline update.