The purpose of this article is to explain the data integration process that DotActiv follows. We answer all the key questions about the various steps of the data integration process.
Applicable Products: DotActiv Pro, DotActiv Enterprise
DotActiv Pro and Enterprise comes standard with an integrated database so that you can have a seamless flow of data to and from your software. This saves you time and improves your accuracy.
What Is Data Integration?
Managing your data manually can become complex and time consuming, and in order to take full advantage of the benefits of having data-driven shelf and floor plans, DotActiv can fully integrate your product, POS (Point of Sales), and market data with the software.
Data integration is the process whereby you pull your retail data (Sales, Stock, Market, and Product) from different sources – POS database, ERP or SAP – and place it into staging tables so that it can be processed through specialised software.
Once your data is available in the software, you will be able to manage it with ease using your data drills and views. You can also choose how you want to drill into your data as well as how you want to view it. From here you can set up and manage your clusters.
You’ll also have the option to have a two-way flow of information, whereby the DotActiv application extracts casual data from planograms when they are published for store implementation.
Data extracted includes product detail information as well as space planning information such as product facings, capacity, minimum display depth, minimum display quantities as required. This information is then available in a SQL export staging table for use to drive replenishment or other business functions.
Our technical support team delivers this database integration service, which comes with a money-back guarantee, capped at the total value of your first years’ billing of the software.
Below is a rough breakdown of the estimated time each phase in the process might take.
Please note that the breakdown below is only a rough estimate and the actual time might differ depending on the client as well as numerous other factors.
The data integration process that DotActiv follows consists of eight steps and needs input from you, the customer.
Step 1: Scoping Meeting
1.1. Why Does DotActiv Insist On Hosting A Scoping Meeting?
The scoping meeting is held to discuss the high-level requirements of the integration project as well as who will be responsible for what.
It is also an opportunity for you to ask DotActiv any questions about the process and to get alignment between everyone involved with regards to the project timeline and expectations.
1.2. What Form Does The Scoping Meeting Take?
The scoping meeting has multiple parts.
Firstly, DotActiv sends you all the necessary documentation. This includes information on the data integration process steps; a Data Feeds document and a Staging Table document.
Data Feeds are a mechanism you would use to transport data from one location to another. An example of a data feed is a Microsoft SQL script.
Secondly, once you have read through all the documentation, you can send back any questions for DotActiv to answer.
Lastly, DotActiv sets up a meeting, usually done online via Google Hangouts, to discuss any final needs.
1.3. What Must The Client Prepare Before The Scoping Meeting Takes Place?
You are expected to read through all the documentation provided to you by DotActiv.
Doing so allows you to educate yourself on the integration process that DotActiv follows. It also means you can ensure that you involve all the necessary people from the beginning.
1.4. Who From The Client’s Side Should Be Present At The Scoping Meeting?
From the clients-side, DotActiv expects the chief business custodian of the DotActiv application to be present.
DotActiv also expects the IT custodian to be present too. That is not necessarily the person doing the work but rather the person who oversees everyone.
While not necessary, DotActiv recommends you include someone from business intelligence as well as someone from infrastructure.
1.5. Who From DotActiv Is Present At The Scoping Meeting?
You can expect a senior IT support technician, to be present who will be your main point of contact during the integration process.
DotActiv includes other authorised IT technicians if and when needed.
1.6. What Does DotActiv Prepare Before The Scoping Meeting Takes Place?
DotActiv prepares any pre-asked questions on the integration process and data feeds where necessary.
Step 2: Deploy The Server
2.1. What Are The Minimum Requirements For The Server?
The minimum requirements for the server are as follows:
- 3Ghz dual-core CPU;
- 16 GB ram; and
- 300 GB HDD.
2.2 What Happens If My Server Doesn’t Meet The Minimum Specs?
You can experience a few issues if your server is under specced. If it is under specced on space, for example, the new data stops importing to your hard drive.
All the automated work also halts because of the hardware limitation.
If the server is under specced on processor or RAM and you have an extensive database with many users simultaneously hitting it, you can expect to experience performance issues.
2.3. What Must I Provide DotActiv With Once I Have Set Up The Server?
DotActiv requests the connection details as well as your company’s security policies.
If DotActiv needs a VPN account to connect to your business, for example, you must provide one along with installation instructions and configuration settings.
The connection details to the server include all the usernames and passwords for the dedicated accounts for DotActiv to complete work.
2.4. What Measures Does DotActiv Have In Place To Ensure My Data Is Safe?
Because the data is on your server, in your environment, and your network, it falls under your security policies.
DotActiv adheres to your security policies. If you give DotActiv a dedicated VPN with full-time access, DotActiv treats it with the utmost respect. Access is only available for authorised users.
If you decide that DotActiv can only get access on request, by VPN with someone supervising, DotActiv will comply.
2.5. Why Does DotActiv Require Permanent Remote Access To My Server?
DotActiv requires permanent remote access for ongoing support purposes. However, long-term, it’s not necessary.
During the first few months, DotActiv actively monitors the integration to ensure it is stable and running smoothly.
Should there be any issues, DotActiv can immediately go onto your server and diagnose the problem.
You can restrict DotActiv’s access to the server if you prefer to go this route. In that case, DotActiv complies with any security policies that you may have in place.
Please note, during the initial implementation, DotActiv requires permanent access because otherwise, the process to get everything up and running is prolonged and difficult.
2.6. Who Is Responsible For Deploying The Server?
Because the server is in your environment and under your control, its deployment is your responsibility.
You are responsible for licensing it, managing it, and maintaining it.
That includes any non-DotActiv related applications such as the operating system, files, anti-virus, anti-malware, SQL, all the hardware, and the network.
DotActiv only maintains the content of the DotActiv databases as well as the DotActiv applications.
2.7. Who Can I Contact At DotActiv If I Have Trouble Deploying My Server?
You need to contact your IT administrator as DotActiv does not carry out hardware support.
DotActiv will offer support when there is a suspicion that the DotActiv software and hardware are not playing well together. In such instances, DotActiv assists as best as possible to work alongside your IT team to diagnose any problem.
2.8. Who At DotActiv Has Access To My Data?
The only person who has access to your data is the IT support consultant/s authorised to access your data for setup and support purposes.
The consultant is the one you met during the scoping meeting.
Outside of that, if you have signed a services contract with DotActiv, it also includes the relevant space planners.
If the initial support consultant is busy when you need assistance, DotActiv assigns the work to the next available support consultant. While that consultant might not have been in the scoping meeting, they have the same authorisation by being part of the support consulting team.
2.9. How Safe And Secure Is My Data If I Give DotActiv Permanent Remote Access To My Server?
Your data is as safe as the security policies you have in place. The data remains on your server, in your environment and network.
DotActiv won’t copy anything off your server unless DotActiv has asked for a local backup for its developers to diagnose a problem.
Even then, DotActiv uses the data to develop a fix for a problem. After that, DotActiv discards the data.
2.10. Can DotActiv Host The Server?
Yes, however, it is always better for you to host the server because it’s your data in your environment and under your scrutiny and security policies.
If you would like DotActiv to assist with hosting your server, then you can read more about this service here.
Step 3: Deploy DotActiv Database
3.1. Who At DotActiv Is Responsible For Deploying The Database?
The authorised IT support consultant agreed upon during the initial scoping meeting is responsible for deploying the DotActiv database.
3.2. Whom Can I Contact If I Have Any Questions About The Deployment Of The Database?
If you believe the deployment of the database is taking too long to complete, your point of contact is the IT support consultant who led the scoping meeting.
If they are busy or on leave, the next person to contact is the salesperson with whom you signed the deal.
3.3. What Is The DotActiv Import Service?
When you script your data into your staging tables, the DotActiv Import Service monitors the table for any new data. If there is new data, it processes that into the DotActiv database structure.
Once this has been completed i.e finished running, it flags it to show it has processed each record. It continuously monitors the data in two-minute cycles.
You can set it to run 24 hours a day or only certain hours, be that during the business day or after hours.
It runs at the time you specify every two minutes to see if there is new data or not. If there is no new data, it then checks the next import.
Data imports include Markets, Products, Sales, Stock, and DC Stock.
3.4. What Database/s Does DotActiv Deploy To My Server?
It is usually one database but it can be up to two.
The primary database is named ‘DA’ and your company name — for example, DARetailerName.
The ‘DA’ is in case the DotActiv database is on a shared server. The naming convention makes it easier for you to identify the DotActv database should there be other DotActiv databases on your server.
If you use StoreComs, you may also have this database on your server.
3.5. What Staging Tables Are Required?
The staging tables needed are documented in the staging tables document, provided before the initial scoping meeting.
The minimum requirements are Markets, Products, and Sales. While not necessary, DotActiv recommends you provide Stock data as well.
3.6. Why Does DotActiv Need To Deploy Its Database To My Server?
DotActiv itself doesn’t need to deploy a database. You could, under instruction, deploy it yourself.
For DotActiv Pro and Enterprise, a database is a requirement. A database is required for integration.
Step 4: Deploy DotActiv Data And Planogram Import Utilities
4.1. What Is The Planogram Import Utility?
The Planogram Import Utility is an application that extracts information from your DotActiv planograms.
The Import Utility extracts the data from your space plans such as product facings, shelves, pegboards, hanging bars, banners, and so on. This information is then imported to your DotActiv Database where it can be reported on.
Your shelf planners will then have a specific folder on their machines where their FLPs will be saved to. The import utility will then push the FLP files to your store communications tool in Activ8, from where your stores can download and implement it.
4.2. What Is The Data Import Utility?
The Data Import Utility works hand in hand with the DotActiv Import Service.
The Import Utility is the front-end application for configuring all imports. It configures which columns in the staging table must go into which value in the DotActiv software.
All these configurations happen in the Data Import Utility, and the Import Service handles the automation thereof.
4.3. What Must I Provide DotActiv With To Ensure Proper Database Deployment?
You must provide DotActiv with administrative rights to complete the installation and set directory permissions.
DotActiv also requires a service account, set to those applications. The server account permanently runs these services.
If you don’t want to provide DotActiv with a password because of security reasons, you can log in and assign that service account to those applications yourself.
Step 5: Populate Staging Tables
5.1. Who Is Responsible For Populating The Staging Tables?
You are responsible for populating the staging tables. Because you pull the data from your systems and push it into the table, it’s also your responsibility to ensure the information is populated accurately and timeously.
You can download staging table documentation here.
5.2. Does DotActiv Assist You With The Setting Up And Maintaining Of Scripts?
When setting up and maintaining scripts, DotActiv does not get involved.
Should you have any questions as to the content of the data and the staging table as well as how it all works, DotActiv can assist.
Also, when it comes to populating the staging tables, DotActiv has the option for you to provide the CSV files in a specific location. DotActiv has scripts that extract it from the location to the staging table.
However, these scripts are pre-defined. DotActiv does not provide scripts to extract data from your systems.
5.3. What Does DotActiv Expect From Me During This Step?
DotActiv expects that you develop the scripts needed, populate the staging tables and to complete the initial testing to ensure that it meets the requirements stipulated in the Data Feeds document.
You must also not alter any of the staging tables as that affects the predefined settings. If you shift data or add it incorrectly, you are expected to correct it.
Also, any information you push into the staging tables is your responsibility. How this data translates into the DotActiv database is DotActiv’s responsibility.
5.4. What Is The Correct Format That My Data Must Be In When Populating The Staging Tables?
You must insert your data into the staging tables as per the predefined data types. This is all stipulated and explained in the Staging Tables document provided before the scoping meeting.
Should you be unable to insert your data directly into the staging tables, we can offer you a script that will process a CSV into the staging table. This CSV would require a predefined naming convention – TableName_Date.CSV, which would go to a predefined location.
5.5. What Happens If The Data Is Not In The Correct Format?
DotActiv runs initial checks to ensure that the data is in the correct format. However, it is not DotActiv’s responsibility to ensure that the data is correct.
If DotActiv finds an issue, it works with your IT team to resolve the issue. A solution includes clearing out your data so that you can re-script it.
If DotActiv finds that the error is because of your negligence, DotActiv can bill for this time at our discretion.
5.6. How Do I Set Up And Maintain My Scripts?
DotActiv doesn’t get involved in setting up or maintaining your scripts.
It is entirely dependant on you, your IT department and your in-house systems.
Step 6: Testing
6.1. What Can You Expect From DotActiv During This Step?
DotActiv conducts preliminary checks on your data. It includes the necessary sense checks to ensure it meets all the requirements.
While these checks ensure that all the data looks correct, it doesn’t guarantee the accuracy of the data within the staging tables.
The next testing takes place after the importation of data into the DotActiv database.
The testing includes looking at the data in the DotActiv database and ensuring it matches what is in your staging tables.
From there, DotActiv also performs basic application functionality. This is to ensure the application works as it should with the database.
6.2. What Does DotActiv Expect From Me During This Step?
DotActiv expects that when you consider the data within the DotActiv application, you agree with it. Are you happy with the data? Do you want to change or update a field?
From there, you must also check the data on your side to ensure its accuracy.
6.3. Can DotActiv Assist Me To Ensure The Server And Data Feeds Are Stable And Accurate?
As the server is in your environment, it is your responsibility to ensure it is both stable and accurate.
You must ensure your data feeds are stable and accurate as DotActiv does not manage your scripts.
You must also be aware of your timings. For example, if you know you script your data on the fourth of every month, you should know that something is wrong if you don’t see new data coming through on the fifth.
You should then contact your IT department to find out if the data did run as expected. If the data didn’t run, you could contact DotActiv to investigate the issue.
If you are unsure of where the problem lies, DotActiv can assist. At the very least, DotActiv can point you in the right direction.
For example, DotActiv can check if there is a new data file or not. If not, you must contact your IT department.
Step 7: Sign Off
7.1. Who Is Responsible For Signing Off On The Integration Process?
Both you and DotActiv are responsible for signing off on the integration process.
When DotActiv sends you the integration hand-over document, which includes all your mappings, validation reports, configuration settings for your user application, you must read through everything.
If you find any issues with it or have any questions, you must raise them.
It is your responsibility to communicate that you’re happy and ready to sign off on the integration process.
7.2. What Critical Points Must Be Checked Off To Complete The Integration Process?
You must ensure all the mappings are correct. The validation report must be signed off, and the user of the application must be able to function and work correctly.
If you find any issues or if you are not happy with anything, you must contact DotActiv as soon as possible so that the problems are corrected.
7.3. What Happens If Any Of The Critical Points Are Missing Or Not Correctly Set-Up?
If any critical points are missing, DotActiv requests that the IT support consultant corrects the work. Should there be a need for escalation, DotActiv will ensure that this happens.
Should an error be found, you don’t have to start the integration process from the beginning.
Sometimes all it requires is a fresh batch of data. If that is the case, DotActiv redeploys everything so you can rerun your data.
Step 8: Go Live
8.1. What Does DotActiv Expect From Me To Complete The Integration Process?
Once you have signed off on the integration, that means you’ve gone live.
DotActiv expects you to treat the system as a live production system from this point forward. It is no longer a test site so you can’t manipulate the data.
If you did attempt to manipulate the data, you could damage it. In that case, you could need to rescript and reprocess your data.
8.2. What Happens If I Come Across Any Issues When Clearing Any Test Data?
If you experience any issues at this stage, you must contact the IT support consultant assigned to your integration so they can resolve it.
Because of the nature of integration, DotActiv allows for a little bit of leeway up until official sign-off. Even after sign-off, there is leeway for the first six months. However, it is within reason.
After that, since you are a fully-fledged production environment, DotActiv is stricter.
8.3. What Can I Expect From DotActiv To Move Forward?
There is no real expectation from DotActiv at this point.
It’s vital to communicate that you have signed off on the integration and are aware that the system is live.
Should you have any questions about the database integration process that DotActiv follows, please contact your Customer Success Consultant.