CI pipeline catches JSON-LD regression
GitHub Actions integration, theme update regression, pre-deploy catch
Context
A development team managing a custom Shopify theme for a mid-market home goods retailer had previously invested time bringing their store's AI readiness score to 61/100. They added Colter as a CI check to protect that investment. During a routine theme update, the CI check caught a regression before it reached production.
Initial scan results
Portfolio average
Can AI agents find your products?
Can AI agents complete purchases?
Is your store safe for automated transactions?
How many AI platforms can interact with your store?
Do AI agents understand your product data?
The workflow
Added Colter to the GitHub Actions pipeline
The team added a colter check step to their existing CI workflow. The check runs against a staging environment on every pull request targeting the main branch. If the composite score drops below the baseline, the CI job fails and the PR is blocked from merging.
Set up a baseline score from the current production state
The team ran colter check against production and saved the result as the baseline. The composite score was 61/100. Per-dimension scores were recorded: Discovery 74, Transaction 45, Security 70, Ecosystem 52, Content Quality 65.
A designer submitted a PR updating the product page template
The PR updated the product detail page layout, reorganized the image gallery, and changed the pricing display. The changes looked correct in the browser preview. No manual tester would have flagged an issue.
CI check ran automatically and failed
The Colter check against staging reported the composite score dropped from 61 to 34. The CI job output showed: Discovery dropped from 74 to 28 (-46 points). All other dimensions were unchanged. The specific finding: JSON-LD Product schema was no longer present on product pages.
Team investigated and identified the root cause
The theme template refactor had moved the product detail section to a new Liquid partial. The JSON-LD script block was in the original template file but was not copied to the new partial. A visual diff of the templates would not have caught this because JSON-LD is rendered in a script tag, invisible in the browser.
Fix applied, CI passed, PR merged safely
The developer added the JSON-LD script block to the new partial. The CI check re-ran and confirmed the score was restored to 61/100. The PR was approved and merged. Total time from regression introduction to fix: under 2 hours.
Key findings
- --Theme template refactor silently removed JSON-LD Product schema
- --The JSON-LD was in a script tag -- invisible in browser previews
- --Visual QA, manual testing, and screenshot comparison would not have caught this
- --Discovery score dropped from 74 to 28 (-46 points) due to a single missing element
- --JSON-LD is the highest-weighted signal in the Discovery dimension
- --Without CI monitoring, the regression would have deployed to production undetected
Remediation steps
Copied the JSON-LD Product schema script block from the old template to the new Liquid partial
Added a code comment marking the JSON-LD block as critical infrastructure not to be removed
Created a team convention to include Colter check results in PR descriptions for visibility
Outcomes
- +Regression caught in CI before reaching production
- +Score preserved at 61/100 -- no customer-facing impact
- +Total time from regression introduction to fix: under 2 hours
- +Team established a convention to flag structural data changes in PR descriptions
- +The CI check now runs on every PR, taking approximately 30 seconds per check
- +No additional tooling cost -- the team uses the free CLI tier
This case study describes a representative workflow based on real data patterns from Colter's batch scan of 363 e-commerce stores. No company names, individuals, or direct quotes are used. Score breakdowns reflect realistic distributions observed in the scan data. This is a pre-launch example, not a customer testimonial.
Run your own audit
Free check, no account required. See your 5-dimension score in under a minute.