Compare commits

..

23 Commits

Author SHA1 Message Date
spouliot da2bb46d5a Tighten Prismatic scrape parsing after live smoke test
Validated against live product pages; fixed three edge cases (also present in
the original JS scraper) surfaced by specialty AkzoNobel products:

- Sample image: only accept real product images on the NIC CDN
  (images.nicindustries.com/prismatic/products), preferring full-size over
  thumbnail. Dropped the loose "prismatic|powder|color" fallback that grabbed
  the site logo on products with no image.
- SDS/TDS/app-guide links: require the href to be an actual document (NIC CDN
  or a .pdf) so a generic /documents nav link isn't captured as the SDS.
- Description: also stop at PRODUCT SUPPORT / PRODUCT COLLECTIONS / CUSTOMER
  SERVICE so less page footer is captured (app-side StripBoilerplate cleans the
  rest).

Structural fields (sku, color, price tiers) verified correct on live data.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 12:41:47 -04:00
spouliot 843d1c3c51 Add token-authenticated catalog import API endpoint
POST /PowderCatalog/ImportApi accepts the JSON scrape format in the request
body, authenticated by a shared secret in the X-Import-Token header (matched
constant-time against CatalogImport:Token), with the vendor in X-Vendor-Name.
Runs through the same ImportJsonAsync -> shared upsert as the manual upload, so
the offline PrismaticSync tool can push unattended.

ImportJsonAsync refactored to take a Stream (the form upload now passes
file.OpenReadStream()). Endpoint is AllowAnonymous + IgnoreAntiforgeryToken
(it's token-gated, not cookie-auth) and returns 401 until a token is configured,
so it's inert by default. README updated with the route + token wiring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 11:35:30 -04:00
spouliot c59d55529f Add PrismaticSync console tool for unattended Prismatic catalog sync
Standalone .NET 8 console app (not part of the main solution) that scrapes the
Prismatic Powders catalog via Playwright and pushes it into the app's catalog
import. Prismatic has no API, so this runs on a workstation (Task Scheduler),
never the deployed server.

- Discovery: incremental newest-first via ?category=created_at (stops once it
  reaches already-known URLs — cheap, finds new colors) and a full all-colors
  crawl for occasional reconcile.
- Scraper: resumable product-page scrape (sku/color/description/price tiers/
  SDS/TDS/app-guide/image), with --refresh-older-than to re-scrape stale
  products and catch price changes. Output matches the app import format so it
  flows through the same shared upsert as the Columbia sync.
- Resilience: brisk randomized base delay, escalating 403 cooldown-and-retry to
  avoid hard bans, periodic rest. All configurable.
- Visibility: streams every product + the inter-product wait to the console
  (colored) and a log file, with an up-front ETA.
- Push: token-authenticated POST to the app import endpoint (skips to manual
  upload when unconfigured).

The app-side token import endpoint is a separate follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 11:30:47 -04:00
spouliot f752abad86 Merge Columbia Coatings catalog integration
Full integration with the Columbia Coatings product catalog API: scheduled +
manual sync of the 2,410-product catalog, multi-manufacturer mapping (Columbia/
PPG/KP Pigments), additives categorization, swatch/tester/sample exclusion, and
tolerant cure/price/HTML parsing. Inventory tie-in (catalog link, discontinued
badge, auto-receive-from-catalog), right-to-delete purge, and quote-at-current-
catalog-price propagation with the cost basis kept separate for accounting.
Shared upsert across API sync and file import, lazy TDS spec enrichment, and
self-healing inventory links. 278 unit tests green.
2026-06-18 08:51:22 -04:00
spouliot 148a3f465e Self-heal inventory catalog links during sync
The sync propagation now also backfills the catalog link: any inventory item
with no PowderCatalogItemId that matches a catalog row by Manufacturer +
ManufacturerPartNumber (the catalog SKU) gets linked and picks up the catalog
price/product data. Only links on a confident match (exact SKU + matching
vendor, or a single unambiguous candidate), so it never mis-links.

This backfills items created before linking existed, automatically, on every
environment (dev and prod) with no manual step or one-off script — legacy items
link on the next sync, new items still link at create time. Cost basis,
quantity, notes, and image remain untouched.

Tests: links an unlinked item by manufacturer+part number; leaves it unlinked
when the part number has no catalog match. Full suite 278 green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 08:32:33 -04:00
spouliot a6538d9638 Add Columbia integration note to help docs
Brief, user-facing mention in both the AI help knowledge base and the human
Inventory help article: the catalog is integrated with the Columbia Coatings
product catalog, refreshes near real-time, auto-fills specs/SDS/TDS, keeps
prices current for quoting, and flags discontinued powders. No API/endpoint
detail, per intent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 16:10:31 -04:00
spouliot 059d94d4fe Lazily enrich catalog specs from TDS on first use
Specific gravity, coverage, and ~55% of cure specs aren't in the Columbia feed.
Rather than read 2,400 TDS PDFs up front, enrich a catalog item the first time
it's actually used:

- FetchTdsCureSpecsAsync now also extracts specific gravity from the TDS.
- New EnsureCatalogTdsSpecsAsync fills a catalog item's specific gravity (and
  any missing cure temp/time) from its TDS, then derives theoretical coverage
  (192.3 / (SG x mils)). No-op once specific gravity is known or when there's no
  TDS; persists to the catalog so the work is done once and benefits everyone.
- Hooked into the catalog->inventory paths (CreateIncomingFromCatalog, the
  custom-powder receive enrichment, and ReceivePowderFromCatalog) so a powder's
  full specs land on both the catalog and the new inventory record. DashboardController
  gains the AI lookup service for this.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 16:07:23 -04:00
spouliot 8401bd77e8 Route Prismatic file import through the shared upsert
The manual JSON file import had its own insert/update loop; it now maps to
PowderCatalogItem and calls IPowderCatalogUpsertService.UpsertAsync — the same
path the Columbia API sync uses — so there is a single upsert/diff implementation
(and the file import now gets inventory propagation for free). Items are tagged
Source = "Manual JSON Import".

Also makes the shared upsert merge-not-wipe: it only overwrites a field when the
incoming feed provides a value (non-blank string, price > 0, nullable HasValue),
so a partial feed like the Prismatic scrape (no cure/chemistry) can't null out
data another source or enrichment populated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 16:01:10 -04:00
spouliot 0f6eef5370 Show current catalog price and price-change nudge on inventory detail
Surfaces the synced CatalogReferencePrice on the inventory detail pricing card:
"Current Catalog Price" (the price quotes use), with an info popover clarifying
it doesn't affect Unit Cost or stock value, an "Updated <date>" line, and a
badge nudging when it differs from what they last paid ("Price up/down from
$X last paid"). Adds CatalogReferencePrice/CatalogPriceUpdatedAt to
InventoryItemDto (auto-mapped). Display only — no pricing/accounting impact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 15:50:11 -04:00
spouliot c22537b68f Propagate catalog price to inventory and quote at current price
Quotes now reflect the current catalog price instead of a tenant's stale
typed-in cost, without disturbing accounting.

- InventoryItem gains CatalogReferencePrice + CatalogPriceUpdatedAt: the
  QUOTING price (current replacement cost), kept separate from UnitCost/
  AverageCost (the cost basis that drives valuation/COGS).
- The catalog sync (PowderCatalogUpsertService.PropagateToLinkedInventoryAsync,
  run at the end of every upsert) refreshes linked inventory items with the
  catalog's current price and product data (description, cure, SDS/TDS, color
  families, coverage, SG, transfer eff, requires-clear-coat). It NEVER touches
  cost, quantity, notes, image, location, or stock levels, and never nulls a
  tenant value with a catalog null. EF persists only actual changes.
- CatalogReferencePrice is also set at link time (catalog receive, incoming-
  from-catalog, identity match on create) so a freshly added powder quotes at
  the current price immediately.
- Pricing now uses CatalogReferencePrice ?? UnitCost: the quote/job powder
  pickers and PricingCalculationService (in-stock usage and powder-to-order
  billing). Falls back to UnitCost for non-catalog/manual powders, so nothing
  regresses. One current price for the whole quantity — no on-hand/to-order
  split. Per-coat snapshot still locks the price at quote creation.

Tests: propagation updates reference price + specs but not cost/qty/notes/
image, and skips a $0 catalog price. Full suite 276 green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 15:34:30 -04:00
spouliot 115ccf7d5e Auto-receive catalog powders and fix soft-deleted SKU collision
Two fixes to the "Got It" powder receive flow:

1. Skip the modal when the powder is in the master catalog. Clicking "Got It"
   now first calls ReceivePowderFromCatalog, which — if the powder resolves in
   the catalog — creates a fully populated inventory record (specs, cure, SDS/
   TDS, image, pricing) and marks the coat received, no modal. Only when the
   powder isn't in the catalog does it fall back to the manual entry modal.
   The catalog match/apply and the receive finalize (opening txn, mark received,
   sibling-coat linking) are extracted into shared helpers used by both the
   modal save and the auto-receive path.

2. Fix a crash re-receiving a previously-deleted powder. The unique index
   IX_InventoryItems_CompanyId_SKU had no filter, so a soft-deleted item still
   reserved its SKU; re-creating it generated the same SKU and violated the
   constraint. The index is now filtered on IsDeleted = 0, matching the app's
   soft-delete semantics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 12:42:29 -04:00
spouliot 99b22d2ad2 Enrich received custom powders from the catalog
When a quote uses a powder the company doesn't stock but the master catalog
does, receiving it ("Got it") created an inventory record with only the color
code/name and cost carried on the quote — cure schedule, SDS/TDS, sample image,
color families, etc. were left blank.

AddCustomPowderToInventory now matches the platform powder catalog by SKU (the
coat's color code, preferring the same manufacturer, then by color name) and
fills every blank spec/document field from it, links PowderCatalogItemId, and
falls back to standard coverage/efficiency defaults. Only gaps are filled, so
anything entered on the receive form is preserved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 12:21:50 -04:00
spouliot 6db055dcf8 Refine Columbia sync after first live run
Fixes from reviewing the first full sync of the real catalog:

- Exclude non-powder / size-variant listings: physical swatch cards (-SW /
  "SWATCH"), 4 oz testers (-04 / "Tester"), and 5 lb sample bags ("Sample (").
  These are not standalone powder colors. Filtered before mapping, and a
  cleanup step deletes any already synced (so they're removed, not flagged
  discontinued). Sample detection keys off the "Sample (" name, not the bare
  -S suffix, to avoid catching a real SKU ending in S (verified 0 collisions).
- Tighten RequiresClearCoat: was flagging ~53% of the catalog on any casual
  "clear coat" mention. Now only genuine signals (partial-cure schedules, the
  Illusion line, explicit "requires a clear" phrasing) trip it.
- Fix literal "&mdash;" in the sync success banner (TempData is HTML-encoded).

Tests cover the exclusion patterns and the tightened clear-coat detection.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 12:01:55 -04:00
spouliot eed61a298b Handle empty featured_image from Columbia feed
Products with no featured image return featured_image: [] (empty array) rather
than an object or null, which failed to bind to a single ColumbiaImage and
aborted the whole sync. Adds ColumbiaImageJsonConverter that reads the object
when present and yields null for any non-object form ([], false, ""), and drops
the unused GalleryImages property (we only use featured_image) to remove the
same risk. Regression tests cover both the empty-array and object cases.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 11:47:28 -04:00
spouliot 2286b5431d Make Columbia API base path configurable
The API namespace (/wp-json/cca/v1) was hardcoded; only the host was in config.
Adds a Columbia:ApiBasePath config key (default /wp-json/cca/v1) so an API
version bump is a config change, not a code change. The client now composes
the products URL from BaseUrl + ApiBasePath + /products. appsettings carries
the live key (private Gitea; Azure App Settings override in prod).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 11:38:32 -04:00
spouliot d2d9f44358 Add Columbia right-to-delete purge action
Phase 5 (part): compliance. PurgeColumbiaData (SuperAdmin) deletes every
catalog record whose Source is the Columbia Coatings API feed — regardless of
derived manufacturer, since PPG and KP Pigments products were served through
that feed — and nulls any inventory PowderCatalogItemId links across all
tenants. Tenant stock records are preserved (they keep their add-time snapshot,
losing only the live catalog link/badge), honoring the boundary that the
distributor's right-to-delete covers their catalog, not customers' purchased
stock. Adds a confirmed "Remove Columbia data" button to the catalog admin.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 11:28:29 -04:00
spouliot 4506c1f641 Link inventory to powder catalog and flag discontinued items
Phase 5 (part): the inventory tie-in.

- Set InventoryItem.PowderCatalogItemId on the catalog-sourced create paths:
  directly in CreateIncomingFromCatalog, and via a new FindCatalogMatchAsync
  (Manufacturer + ManufacturerPartNumber) helper in Create.
- Inventory Details loads the linked catalog row (falling back to an identity
  match for items created before linking) and shows a "Discontinued by
  manufacturer — cannot reorder" badge + banner when it's discontinued.
  Deliberately distinct from the shop's own Active/Inactive status: existing
  stock can still be used and quoted, it just can't be reordered.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 11:18:15 -04:00
spouliot a07f6aa1a8 Add scheduled Columbia catalog sync background service
Phase 4: automation. ColumbiaCatalogSyncBackgroundService wakes hourly and
runs a full sync only when ColumbiaSyncEnabled is on and ColumbiaSyncIntervalDays
has elapsed since the last successful run (tracked via the ColumbiaLastSyncedAt
setting). No-ops quietly when disabled or unconfigured. The hourly due-check is
negligible; the actual sync runs at most once per interval. Sync failures are
recorded on the result/settings, never thrown, so a bad run can't kill the loop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 11:12:27 -04:00
spouliot 9aa3a99488 Add manual "Sync Columbia" action and status to powder catalog admin
Phase 3: SuperAdmin-triggered sync. Adds a SyncColumbia POST action that runs
a full catalog sync on demand (bypassing the schedule) and reports the result
via TempData. The catalog index header gains a "Sync Columbia" button (with a
syncing spinner) and a status line showing the scheduled-sync on/off state,
last-synced time, and last-run summary, read from the platform settings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 11:05:21 -04:00
spouliot 2b420d4623 Add Columbia catalog mapper, shared upsert, and sync service
Phase 2: the mapping and sync core.

- ColumbiaCatalogMapper (pure/static, unit-tested): maps an API product to a
  PowderCatalogItem. Derives manufacturer (PPG/KP Pigments/Columbia) from
  taxonomy+SKU; flags additives into the Powder Additives category; takes base
  price from the top-level price with variant fallback; captures variation /
  tiered pricing as JSON; parses the free-text cure schedule into all curves
  (three degree glyphs, @/at, multi-curve in order, partial-cure -> none) with
  the first as the primary temp/time; strips HTML descriptions; joins color
  groups; normalizes chemistry; flags clear-coat powders.

- PowderCatalogUpsertService (IPowderCatalogUpsertService): single upsert path
  matching on (VendorName, SKU). Copies only feed-sourced fields and leaves
  enrichment fields (specific gravity, coverage, transfer efficiency, finish)
  untouched so syncs never wipe lazily-enriched TDS/AI data.

- ColumbiaCatalogSyncService (IColumbiaCatalogSyncService): pulls the full
  catalog, maps + de-dupes, upserts, then reconciles discontinuations ONLY on a
  complete pull (a partial pull throws and aborts before the sweep). Reactivates
  reappearing items; records last-synced/last-result platform settings.

- 25 mapper unit tests covering the cure parser, manufacturer derivation,
  simple/variable pricing, chemistry, color, and HTML cases from real records.
  Full suite green (261 passed).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 11:02:12 -04:00
spouliot a4a3dde7e4 Add single-product lookup methods to Columbia API client
Adds GetProductBySkuAsync (GET /products?sku=) and GetProductByIdAsync
(GET /products/{id}) for ad-hoc / on-demand refresh of a single record
without pulling the full catalog. Extracts the shared 429-retry send loop
into SendWithRetryAsync, which now also treats 404 as not-found (null) so
single lookups don't throw on a missing SKU/ID.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 10:51:02 -04:00
spouliot 39f61b9718 Add Columbia Coatings API client, DTOs, and sync settings
Phase 1b of the Columbia Coatings integration: the typed read client and
its configuration, ahead of the sync/mapper service.

- ColumbiaProductDtos: wire-shape models for GET /products. tiered_pricing
  is captured as JsonElement because the API returns it as an object on
  simple products but an empty array on variable ones — binding it raw
  avoids a deserialization throw; the mapper interprets it.
- IColumbiaCoatingsApiClient / ColumbiaCoatingsApiClient: pages the catalog
  via GET /products (NOT the export download_url, which is Cloudflare-blocked
  for server clients). Sends X-API-Key from config, honors 429/Retry-After,
  and THROWS on any page failure so a partial pull can never be mistaken for
  the full catalog (protects the later discontinuation sweep).
- ColumbiaIntegrationConstants: single home for config keys, setting keys,
  and the derived Source/manufacturer/category values.
- Config: Columbia:ApiKey (blank — secret supplied per environment) and
  Columbia:BaseUrl in appsettings.
- SeedColumbiaSyncSettings migration: seeds SuperAdmin-managed platform
  settings ColumbiaSyncEnabled (off by default), ColumbiaSyncIntervalDays
  (7), and last-sync tracking, under a new "Integrations" group.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 10:47:00 -04:00
spouliot c98f9faf63 Add Columbia Coatings catalog integration schema fields
Phase 1a of the Columbia Coatings API integration. Adds the persisted
fields the sync/mapper will need, ahead of the client and sync service:

PowderCatalogItem:
- Category: our product category (e.g. "Powder Additives" for gram-sold
  pigments) derived from vendor taxonomy at import, not stored raw.
- Source: provenance (e.g. "Columbia Coatings API"), kept separate from
  VendorName (= derived manufacturer) so a distributor's right-to-delete
  can purge by feed regardless of manufacturer.
- ChemistryType: resin chemistry (Polyester/TGIC/Epoxy/...), distinct
  from Finish.
- MilThickness: recommended film build as vendor free text.
- CureScheduleText: raw cure schedule verbatim (formats vary widely).
- CureCurvesJson: all parsed cure curves, so alternate low-temp curves
  are preserved for heat-sensitive substrates, not just the primary.
- FormulationChanges: vendor reformulation log; a signal cure specs may
  have changed.

InventoryItem:
- PowderCatalogItemId: loose link to the catalog row (matches the
  QuoteItemCoat pattern) so inventory detail can show manufacturer-level
  status (e.g. discontinued/cannot reorder) and future change flags.
  Nulled, never cascaded, when source catalog data is purged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 10:37:56 -04:00
54 changed files with 49797 additions and 131 deletions
@@ -0,0 +1,8 @@
# Build output
bin/
obj/
# Transient scrape artifacts
*.tmp
*.invalid-*.bak
prismatic-sync.log
@@ -0,0 +1,43 @@
using Microsoft.Playwright;
namespace PrismaticSync.Infrastructure;
/// <summary>
/// A headless Chromium session with a realistic desktop fingerprint (UA, viewport, locale,
/// timezone) — matching the original scraper's settings to look like a normal browser.
/// </summary>
public sealed class BrowserSession : IAsyncDisposable
{
private IPlaywright? _pw;
private IBrowser? _browser;
private IBrowserContext? _context;
public IPage Page { get; private set; } = null!;
public static async Task<BrowserSession> CreateAsync(bool headed)
{
var session = new BrowserSession();
session._pw = await Playwright.CreateAsync();
session._browser = await session._pw.Chromium.LaunchAsync(new BrowserTypeLaunchOptions
{
Headless = !headed
});
session._context = await session._browser.NewContextAsync(new BrowserNewContextOptions
{
UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " +
"(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
ViewportSize = new ViewportSize { Width = 1365, Height = 900 },
Locale = "en-US",
TimezoneId = "America/New_York"
});
session.Page = await session._context.NewPageAsync();
return session;
}
public async ValueTask DisposeAsync()
{
if (_context is not null) await _context.CloseAsync();
if (_browser is not null) await _browser.CloseAsync();
_pw?.Dispose();
}
}
@@ -0,0 +1,65 @@
using System.Text.Json;
using PrismaticSync.Models;
namespace PrismaticSync.Infrastructure;
/// <summary>Loads/saves the scrape output and the URL list, with atomic writes so a crash mid-save can't corrupt them.</summary>
public static class JsonStore
{
private static readonly JsonSerializerOptions WriteOptions = new() { WriteIndented = true };
private static readonly JsonSerializerOptions ReadOptions = new() { PropertyNameCaseInsensitive = true };
public static ScrapeOutput LoadOutput(string path)
{
if (!File.Exists(path))
return new ScrapeOutput();
var json = File.ReadAllText(path);
try
{
// Tolerate a bare array (older output format) as well as { results, errors }.
if (json.TrimStart().StartsWith("["))
{
var results = JsonSerializer.Deserialize<List<ProductRecord>>(json, ReadOptions) ?? new();
return new ScrapeOutput { Results = results };
}
return JsonSerializer.Deserialize<ScrapeOutput>(json, ReadOptions) ?? new ScrapeOutput();
}
catch (Exception ex)
{
var backup = $"{path}.invalid-{DateTimeOffset.UtcNow.ToUnixTimeSeconds()}.bak";
File.Copy(path, backup, overwrite: true);
throw new InvalidOperationException($"Could not parse {path}. Backed it up to {backup}. {ex.Message}");
}
}
public static void SaveOutput(string path, ScrapeOutput data)
{
var tmp = path + ".tmp";
File.WriteAllText(tmp, JsonSerializer.Serialize(data, WriteOptions));
File.Move(tmp, path, overwrite: true);
}
public static List<string> LoadUrls(string path)
{
if (!File.Exists(path))
return new List<string>();
return File.ReadAllLines(path)
.Select(CleanUrl)
.Where(u => u.Length > 0 && !u.StartsWith("#"))
.Distinct(StringComparer.OrdinalIgnoreCase)
.ToList();
}
public static void SaveUrls(string path, IEnumerable<string> urls)
{
var sorted = urls.Distinct(StringComparer.OrdinalIgnoreCase).OrderBy(u => u, StringComparer.OrdinalIgnoreCase);
var tmp = path + ".tmp";
File.WriteAllText(tmp, string.Join(Environment.NewLine, sorted) + Environment.NewLine);
File.Move(tmp, path, overwrite: true);
}
public static string CleanUrl(string? url) =>
(url ?? string.Empty).Split('?')[0].Split('#')[0].Trim();
}
@@ -0,0 +1,49 @@
namespace PrismaticSync.Infrastructure;
/// <summary>
/// Minimal timestamped logger — writes to the console and appends to a rolling log file so an
/// unattended (Task Scheduler) run leaves an audit trail. Intentionally dependency-free.
/// </summary>
public static class Log
{
private static string _logFile = "prismatic-sync.log";
private static readonly object Gate = new();
public static void Configure(string logFile) => _logFile = logFile;
public static void Info(string message) => Write("INFO", message);
public static void Warn(string message) => Write("WARN", message);
public static void Error(string message) => Write("ERROR", message);
private static void Write(string level, string message)
{
var line = $"[{DateTime.UtcNow:yyyy-MM-ddTHH:mm:ssZ}] {level,-5} {message}";
// Live console stream (visible on a manual run); color-code so warnings/errors stand out.
lock (Gate)
{
var color = level switch
{
"WARN" => ConsoleColor.Yellow,
"ERROR" => ConsoleColor.Red,
_ => (ConsoleColor?)null
};
if (color is { } c)
{
var previous = Console.ForegroundColor;
Console.ForegroundColor = c;
Console.WriteLine(line);
Console.ForegroundColor = previous;
}
else
{
Console.WriteLine(line);
}
// File trail — never let logging break a run.
try { File.AppendAllText(_logFile, line + Environment.NewLine); }
catch { /* ignore */ }
}
}
}
@@ -0,0 +1,69 @@
namespace PrismaticSync.Infrastructure;
/// <summary>Strongly-typed config bound from the "Sync" section of appsettings.json.</summary>
public class SyncConfig
{
public string BaseUrl { get; set; } = "https://www.prismaticpowders.com";
public string ColorsPath { get; set; } = "/shop/powder-coating-colors";
public string ProductUrlsFile { get; set; } = "product-urls.txt";
public string OutputJsonFile { get; set; } = "prismatic_powders.json";
public string LogFile { get; set; } = "prismatic-sync.log";
/// <summary>Politeness delay between product scrapes (randomized within the range).</summary>
public int MinDelaySeconds { get; set; } = 6;
public int MaxDelaySeconds { get; set; } = 14;
/// <summary>On a 403/block, cool down this many seconds × the consecutive-block count, then retry.</summary>
public int BlockedCooldownSeconds { get; set; } = 120;
/// <summary>Upper bound on a single cooldown so escalation can't run away.</summary>
public int BlockedCooldownMaxSeconds { get; set; } = 600;
/// <summary>How many times to cool-down-and-retry a blocked product before recording it as an error.</summary>
public int BlockedMaxRetries { get; set; } = 3;
/// <summary>Take a longer rest after this many products (0 disables). Eases load and looks less robotic.</summary>
public int LongRestEveryProducts { get; set; } = 150;
/// <summary>Length of the periodic long rest, in seconds.</summary>
public int LongRestSeconds { get; set; } = 45;
/// <summary>Extra settle time after a product page loads before reading it.</summary>
public int PageSettleSeconds { get; set; } = 4;
/// <summary>Pause after each scroll while a listing lazy-loads more items.</summary>
public int ScrollWaitMs { get; set; } = 1500;
/// <summary>Hard cap on scrolls per listing, as a safety stop.</summary>
public int MaxScrolls { get; set; } = 400;
/// <summary>Full discovery: stop a listing after this many scrolls add no new links.</summary>
public int StopAfterNoNewScrolls { get; set; } = 10;
/// <summary>
/// Incremental discovery: stop the newest-first listing after this many consecutive scrolls
/// that surfaced only already-known URLs — i.e. we've scrolled past the new products.
/// </summary>
public int StopAfterKnownScrolls { get; set; } = 8;
/// <summary>Color filter params used by full discovery.</summary>
public string[] ColorParams { get; set; } = Array.Empty<string>();
public ImportConfig Import { get; set; } = new();
public string ColorsUrl => $"{BaseUrl.TrimEnd('/')}{ColorsPath}";
}
/// <summary>Where and how to push the scraped catalog into the app.</summary>
public class ImportConfig
{
/// <summary>Full URL of the app's token-authenticated catalog import endpoint.</summary>
public string EndpointUrl { get; set; } = "";
/// <summary>Shared secret sent in the X-Import-Token header. Must match the app's config.</summary>
public string Token { get; set; } = "";
/// <summary>Vendor name applied to every record on import.</summary>
public string VendorName { get; set; } = "Prismatic Powders";
}
@@ -0,0 +1,45 @@
using System.Text.Json.Serialization;
namespace PrismaticSync.Models;
/// <summary>
/// On-disk scrape output. Shape matches the app's catalog import (a top-level "results" array of
/// snake_case product records), so the JSON drops straight into the import endpoint. "errors" tracks
/// failed URLs for resumable re-runs.
/// </summary>
public class ScrapeOutput
{
[JsonPropertyName("results")] public List<ProductRecord> Results { get; set; } = new();
[JsonPropertyName("errors")] public List<ScrapeError> Errors { get; set; } = new();
}
/// <summary>One scraped product, in the import's expected field shape.</summary>
public class ProductRecord
{
[JsonPropertyName("sku")] public string Sku { get; set; } = "";
[JsonPropertyName("color_name")] public string ColorName { get; set; } = "";
[JsonPropertyName("description")] public string Description { get; set; } = "";
[JsonPropertyName("price_tiers")] public List<PriceTier> PriceTiers { get; set; } = new();
[JsonPropertyName("safety_data_sheet_url")] public string SafetyDataSheetUrl { get; set; } = "";
[JsonPropertyName("technical_data_sheet_url")] public string TechnicalDataSheetUrl { get; set; } = "";
[JsonPropertyName("application_guide_url")] public string ApplicationGuideUrl { get; set; } = "";
[JsonPropertyName("sample_image_url")] public string SampleImageUrl { get; set; } = "";
[JsonPropertyName("product_url")] public string ProductUrl { get; set; } = "";
[JsonPropertyName("scraped_at")] public DateTime ScrapedAt { get; set; }
}
/// <summary>A quantity-break price tier — {min, max, price}. max is null for an open-ended top tier.</summary>
public class PriceTier
{
[JsonPropertyName("min")] public int? Min { get; set; }
[JsonPropertyName("max")] public int? Max { get; set; }
[JsonPropertyName("price")] public decimal Price { get; set; }
}
/// <summary>A URL that failed to scrape, kept so resumable runs can skip or retry it.</summary>
public class ScrapeError
{
[JsonPropertyName("product_url")] public string ProductUrl { get; set; } = "";
[JsonPropertyName("error")] public string Error { get; set; } = "";
[JsonPropertyName("scraped_at")] public DateTime ScrapedAt { get; set; }
}
@@ -0,0 +1,36 @@
<Project Sdk="Microsoft.NET.Sdk">
<!--
Standalone workstation tool — deliberately NOT part of PowderCoating.sln.
Build/publish independently and run on a machine you control (Task Scheduler),
never on the deployed app server. Scrapes Prismatic Powders and pushes the
result into the app's catalog import endpoint.
First-time setup on a workstation:
dotnet build
pwsh bin/Debug/net8.0/playwright.ps1 install chromium
-->
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net8.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AssemblyName>PrismaticSync</AssemblyName>
<RootNamespace>PrismaticSync</RootNamespace>
<InvariantGlobalization>true</InvariantGlobalization>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Playwright" Version="1.49.0" />
<PackageReference Include="Microsoft.Extensions.Configuration" Version="8.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="8.0.1" />
<PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="8.0.2" />
</ItemGroup>
<ItemGroup>
<None Update="appsettings.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>
+106
View File
@@ -0,0 +1,106 @@
using Microsoft.Extensions.Configuration;
using PrismaticSync.Infrastructure;
using PrismaticSync.Services;
// ── Load config ───────────────────────────────────────────────────────────────
var configRoot = new ConfigurationBuilder()
.SetBasePath(AppContext.BaseDirectory)
.AddJsonFile("appsettings.json", optional: false)
.Build();
var config = configRoot.GetSection("Sync").Get<SyncConfig>() ?? new SyncConfig();
Log.Configure(config.LogFile);
// ── Parse args ────────────────────────────────────────────────────────────────
var command = args.Length > 0 && !args[0].StartsWith("--") ? args[0].ToLowerInvariant() : "run";
var headed = args.Contains("--headed");
var retryErrors = args.Contains("--retry-errors");
var maxProducts = GetIntArg("--max-products", 0);
// "run" refreshes products older than 30 days by default; explicit commands default to new-only.
var refreshOlderThanDays = GetIntArg("--refresh-older-than", command == "run" ? 30 : 0);
Log.Info($"PrismaticSync — command '{command}' (headed={headed}, refreshOlderThan={refreshOlderThanDays}d, maxProducts={maxProducts})");
try
{
switch (command)
{
case "discover-new":
await WithBrowser(d => new PrismaticDiscoverer(d, config).DiscoverNewAsync());
break;
case "discover-full":
await WithBrowser(d => new PrismaticDiscoverer(d, config).DiscoverFullAsync());
break;
case "scrape":
await WithBrowser(d => new PrismaticScraper(d, config).ScrapeAsync(refreshOlderThanDays, maxProducts, retryErrors));
break;
case "push":
await new CatalogPusher(config).PushAsync();
break;
case "run":
// The scheduled default: find new colors, scrape new + stale, then push.
await WithBrowser(async d =>
{
await new PrismaticDiscoverer(d, config).DiscoverNewAsync();
await new PrismaticScraper(d, config).ScrapeAsync(refreshOlderThanDays, maxProducts, retryErrors);
});
await new CatalogPusher(config).PushAsync();
break;
default:
PrintUsage();
return 1;
}
Log.Info("Done.");
return 0;
}
catch (Exception ex)
{
Log.Error($"Fatal: {ex}");
return 1;
}
// ── Helpers ───────────────────────────────────────────────────────────────────
async Task WithBrowser(Func<BrowserSession, Task> action)
{
await using var session = await BrowserSession.CreateAsync(headed);
await action(session);
}
int GetIntArg(string name, int fallback)
{
var prefix = name + "=";
var found = args.FirstOrDefault(a => a.StartsWith(prefix, StringComparison.OrdinalIgnoreCase));
return found is not null && int.TryParse(found[prefix.Length..], out var value) ? value : fallback;
}
void PrintUsage()
{
Console.WriteLine(
"""
PrismaticSync scrape Prismatic Powders and push to the app catalog.
Usage: PrismaticSync [command] [options]
Commands:
run (default) discover-new + scrape (new + stale) + push
discover-new Incremental discovery via newest-first sort (cheap; finds new colors)
discover-full Full discovery across all color filters (heavy; reconciles the whole set)
scrape Scrape product pages from the URL list (resumable)
push Push the scraped JSON to the import endpoint
Options:
--refresh-older-than=N Re-scrape products whose data is older than N days (default 30 for 'run')
--max-products=N Cap products scraped this run (0 = no cap)
--retry-errors Retry URLs previously recorded as errors
--headed Show the browser window (debugging)
Config: appsettings.json (delays, file paths, import endpoint + token).
First run on a new machine: dotnet build, then `pwsh bin/Debug/net8.0/playwright.ps1 install chromium`.
""");
}
+86
View File
@@ -0,0 +1,86 @@
# PrismaticSync
A standalone .NET console tool that scrapes the Prismatic Powders catalog and pushes it into the
Powder Coating Logix catalog import endpoint. It exists because Prismatic has **no API** (unlike
Columbia Coatings) — so the data has to be scraped via browser automation.
> **Runs on a workstation you control — never on the deployed app server.** Scraping from the cloud
> app's IP would get blocked and isn't appropriate. This tool is deliberately *not* part of
> `PowderCoating.sln`; build and run it independently.
## First-time setup (per machine)
```powershell
cd "scripts/Prismatic Data Scraper"
dotnet build
pwsh bin/Debug/net8.0/playwright.ps1 install chromium # one-time browser download
```
## Commands
```powershell
dotnet run -- run # default: discover-new + scrape (new + stale >30d) + push
dotnet run -- discover-new # cheap: find newly-added colors (newest-first, stops at known)
dotnet run -- discover-full # heavy: crawl all color filters (reconcile whole set / removals)
dotnet run -- scrape # scrape product pages from product-urls.txt (resumable)
dotnet run -- scrape --refresh-older-than=30 # also re-scrape products older than 30 days (price changes)
dotnet run -- push # push prismatic_powders.json to the import endpoint
```
Options: `--max-products=N`, `--retry-errors`, `--headed` (show the browser for debugging).
Everything streams to the console live (warnings/errors in color) **and** to `prismatic-sync.log`.
## Operating model (suggested cadence)
| Run | Command | Cadence | Why |
|-----|---------|---------|-----|
| Find new colors | `run` (does discover-new + scrape-new) | Weekly | Cheap; Prismatic adds colors often |
| Price refresh | `scrape --refresh-older-than=30` then `push` | Monthly | Re-scrapes stale products to catch price changes (slow, ~hours) |
| Full reconcile | `discover-full` then `scrape` | Quarterly | Catches removed/discontinued colors |
A full scrape of ~5,000 products takes hours (polite delays). It saves after every product and is
fully resumable, so stop/restart any time.
## Politeness / anti-block
Configurable in `appsettings.json`: randomized 614s base delay, an escalating **cooldown + retry on
403** (so a temporary block doesn't get you hard-banned mid-run), and a periodic long rest. Leave
these conservative — getting blocked is worse than being slow, and Prismatic is a partner.
## Pushing into the app
Set in `appsettings.json`:
- `Sync.Import.EndpointUrl``https://<your-app>/PowderCatalog/ImportApi`
- `Sync.Import.Token` → the same secret as the app's `CatalogImport:Token` config
The tool POSTs the JSON with an `X-Import-Token` header (and `X-Vendor-Name: Prismatic Powders`) to
that endpoint, which authenticates the token and runs the records through the same upsert as the
Columbia sync. If the endpoint/token isn't configured here, `push` is skipped and you upload
`prismatic_powders.json` manually via the Powder Catalog admin page instead.
> **App side:** set `CatalogImport:Token` in the web app's config (Azure App Setting in prod). The
> endpoint returns 401 until a token is set, so it's inert by default.
## Scheduling (Windows Task Scheduler)
Point a scheduled task at the published exe (or `dotnet run`). Example weekly task command:
```
Program/script: C:\Tools\PrismaticSync\PrismaticSync.exe
Arguments: run
Start in: C:\Tools\PrismaticSync
```
Publish a self-contained build to drop on the workstation:
```powershell
dotnet publish -c Release -r win-x64 --self-contained false -o C:\Tools\PrismaticSync
pwsh C:\Tools\PrismaticSync\playwright.ps1 install chromium
```
## The long game
This is the interim path. The durable endgame is a real Prismatic **API** (the partnership), at which
point this tool is replaced by a clean in-app sync like Columbia's — reusing the same upsert,
propagation, and discontinued handling.
@@ -0,0 +1,63 @@
using System.Text;
using PrismaticSync.Infrastructure;
namespace PrismaticSync.Services;
/// <summary>
/// Pushes the scraped JSON to the app's token-authenticated catalog import endpoint. When no
/// endpoint is configured it no-ops (the JSON is still on disk for a manual upload), so the tool is
/// useful before the endpoint exists.
/// </summary>
public class CatalogPusher
{
private readonly SyncConfig _config;
public CatalogPusher(SyncConfig config) => _config = config;
public async Task<bool> PushAsync()
{
if (string.IsNullOrWhiteSpace(_config.Import.EndpointUrl))
{
Log.Warn($"No import endpoint configured (Sync.Import.EndpointUrl) — skipping push. " +
$"Upload {_config.OutputJsonFile} manually via the Powder Catalog admin instead.");
return false;
}
if (!File.Exists(_config.OutputJsonFile))
{
Log.Warn($"Output file {_config.OutputJsonFile} not found — nothing to push.");
return false;
}
var json = await File.ReadAllTextAsync(_config.OutputJsonFile);
Log.Info($"Pushing {_config.OutputJsonFile} to {_config.Import.EndpointUrl} (vendor: {_config.Import.VendorName})...");
using var http = new HttpClient { Timeout = TimeSpan.FromMinutes(5) };
using var request = new HttpRequestMessage(HttpMethod.Post, _config.Import.EndpointUrl);
request.Headers.Add("X-Import-Token", _config.Import.Token);
request.Headers.Add("X-Vendor-Name", _config.Import.VendorName);
request.Content = new StringContent(json, Encoding.UTF8, "application/json");
try
{
using var response = await http.SendAsync(request);
var body = await response.Content.ReadAsStringAsync();
if (response.IsSuccessStatusCode)
{
Log.Info($"Push succeeded ({(int)response.StatusCode}): {Trim(body)}");
return true;
}
Log.Error($"Push failed ({(int)response.StatusCode}): {Trim(body)}");
return false;
}
catch (Exception ex)
{
Log.Error($"Push error: {ex.Message}");
return false;
}
}
private static string Trim(string s) => s.Length > 500 ? s[..500] + "…" : s;
}
@@ -0,0 +1,138 @@
using System.Text.RegularExpressions;
using Microsoft.Playwright;
using PrismaticSync.Infrastructure;
namespace PrismaticSync.Services;
/// <summary>
/// Discovers product URLs from the Prismatic color listing (infinite-scroll). Two modes:
/// incremental (newest-first via <c>?category=created_at</c>, stop once we reach already-known
/// URLs) for cheap frequent runs, and full (every color filter to the bottom) for occasional
/// reconciliation. Both append to the URL list file.
/// </summary>
public class PrismaticDiscoverer
{
private static readonly Regex ProductUrlRegex =
new(@"/shop/powder-coating-colors/[A-Z0-9-]+/", RegexOptions.IgnoreCase | RegexOptions.Compiled);
private readonly BrowserSession _session;
private readonly SyncConfig _config;
public PrismaticDiscoverer(BrowserSession session, SyncConfig config)
{
_session = session;
_config = config;
}
/// <summary>
/// Incremental discovery: crawl the newest-first listing and stop once a run of consecutive
/// scrolls surfaces only already-known URLs — meaning we've scrolled past the new products.
/// Returns the count of newly found URLs.
/// </summary>
public async Task<int> DiscoverNewAsync()
{
var known = new HashSet<string>(JsonStore.LoadUrls(_config.ProductUrlsFile), StringComparer.OrdinalIgnoreCase);
var startCount = known.Count;
Log.Info($"Incremental discovery (newest first). Known URLs: {startCount}");
await GotoAsync($"{_config.ColorsUrl}?category=created_at");
var knownStreak = 0;
for (var i = 0; i < _config.MaxScrolls; i++)
{
var addedNew = 0;
foreach (var link in await CollectProductLinksAsync())
if (known.Add(link)) addedNew++;
JsonStore.SaveUrls(_config.ProductUrlsFile, known);
knownStreak = addedNew == 0 ? knownStreak + 1 : 0;
Log.Info($"Scroll {i + 1}: +{addedNew} new, total {known.Count}, known-streak {knownStreak}");
if (knownStreak >= _config.StopAfterKnownScrolls)
{
Log.Info("Reached known territory — stopping incremental discovery.");
break;
}
await ScrollAsync();
}
var newCount = known.Count - startCount;
Log.Info($"Incremental discovery done. New URLs: {newCount}; total {known.Count}");
return newCount;
}
/// <summary>
/// Full discovery: crawl every color filter to the bottom. Heavier — use occasionally to
/// reconcile the whole set (e.g. to notice colors that have been removed). Returns new URL count.
/// </summary>
public async Task<int> DiscoverFullAsync()
{
var known = new HashSet<string>(JsonStore.LoadUrls(_config.ProductUrlsFile), StringComparer.OrdinalIgnoreCase);
var startCount = known.Count;
Log.Info($"Full discovery across {_config.ColorParams.Length} color filters. Known URLs: {startCount}");
foreach (var color in _config.ColorParams)
{
Log.Info($"Color filter: {color}");
try
{
await GotoAsync($"{_config.ColorsUrl}?color={Uri.EscapeDataString(color)}");
var noNew = 0;
for (var i = 0; i < _config.MaxScrolls; i++)
{
var added = 0;
foreach (var link in await CollectProductLinksAsync())
if (known.Add(link)) added++;
JsonStore.SaveUrls(_config.ProductUrlsFile, known);
noNew = added == 0 ? noNew + 1 : 0;
if (noNew >= _config.StopAfterNoNewScrolls)
break;
await ScrollAsync();
}
Log.Info($"Color {color} done. Total {known.Count}");
await _session.Page.WaitForTimeoutAsync(3000);
}
catch (Exception ex)
{
Log.Warn($"Color {color} failed: {ex.Message}");
}
}
var newCount = known.Count - startCount;
Log.Info($"Full discovery done. New this run: {newCount}; total {known.Count}");
return newCount;
}
private async Task GotoAsync(string url)
{
await _session.Page.GotoAsync(url, new PageGotoOptions
{
WaitUntil = WaitUntilState.DOMContentLoaded,
Timeout = 60000
});
await _session.Page.WaitForTimeoutAsync(_config.PageSettleSeconds * 1000);
}
private async Task ScrollAsync()
{
await _session.Page.Mouse.WheelAsync(0, 2500);
await _session.Page.WaitForTimeoutAsync(_config.ScrollWaitMs);
}
private async Task<List<string>> CollectProductLinksAsync()
{
var hrefs = await _session.Page.EvalOnSelectorAllAsync<string[]>(
"a", "els => els.map(a => a.href).filter(Boolean)");
return hrefs
.Where(h => ProductUrlRegex.IsMatch(h))
.Select(JsonStore.CleanUrl)
.Where(u => u.Length > 0)
.ToList();
}
}
@@ -0,0 +1,308 @@
using System.Diagnostics;
using System.Globalization;
using System.Text.RegularExpressions;
using Microsoft.Playwright;
using PrismaticSync.Infrastructure;
using PrismaticSync.Models;
namespace PrismaticSync.Services;
/// <summary>
/// Scrapes individual Prismatic product pages into <see cref="ProductRecord"/>s. Resumable (skips
/// already-scraped URLs, optionally retries past errors) and supports a refresh window so stale
/// records get re-scraped to catch price changes. Saves after every product so a long run can be
/// stopped and resumed safely, and logs continuously — including the delay between products — so a
/// manual run always shows it's alive.
/// </summary>
public class PrismaticScraper
{
private static readonly Regex ProductUrlRegex =
new(@"/shop/powder-coating-colors/[A-Z0-9-]+/", RegexOptions.IgnoreCase | RegexOptions.Compiled);
private static readonly Regex SkuRegex =
new(@"Item:\s*([A-Z0-9-]+)", RegexOptions.IgnoreCase | RegexOptions.Compiled);
private static readonly Regex DescRegex =
new(@"Description:\s*(.*?)(WARNING:|What does this match\?|PRODUCT SUPPORT|PRODUCT COLLECTIONS|CUSTOMER SERVICE|$)",
RegexOptions.IgnoreCase | RegexOptions.Singleline | RegexOptions.Compiled);
private static readonly Regex PriceTierRegex =
new(@"(\d+\s*-\s*\d+\s*lbs|\d+\s*\+\s*lbs)\s*\$([\d.]+)", RegexOptions.IgnoreCase | RegexOptions.Compiled);
private static readonly Regex RangeRegex = new(@"(\d+)\s*-\s*(\d+)", RegexOptions.Compiled);
private static readonly Regex PlusRegex = new(@"(\d+)\s*\+", RegexOptions.Compiled);
private static readonly Regex WhitespaceRegex = new(@"\s+", RegexOptions.Compiled);
private readonly BrowserSession _session;
private readonly SyncConfig _config;
private readonly Random _random = new();
public PrismaticScraper(BrowserSession session, SyncConfig config)
{
_session = session;
_config = config;
}
/// <summary>
/// Scrapes products needing work: those not yet scraped, plus (when <paramref name="refreshOlderThanDays"/>
/// &gt; 0) any whose data is older than that window. Returns (scraped, errors).
/// </summary>
public async Task<(int Scraped, int Errors)> ScrapeAsync(int refreshOlderThanDays, int maxProducts, bool retryErrors)
{
var allUrls = JsonStore.LoadUrls(_config.ProductUrlsFile)
.Where(u => ProductUrlRegex.IsMatch(u))
.ToList();
var data = JsonStore.LoadOutput(_config.OutputJsonFile);
// Index existing results by URL (keep the most recent if the file has dupes).
var resultByUrl = data.Results
.GroupBy(r => JsonStore.CleanUrl(r.ProductUrl), StringComparer.OrdinalIgnoreCase)
.ToDictionary(g => g.Key, g => g.OrderByDescending(r => r.ScrapedAt).First(), StringComparer.OrdinalIgnoreCase);
var errorUrls = new HashSet<string>(
data.Errors.Select(e => JsonStore.CleanUrl(e.ProductUrl)), StringComparer.OrdinalIgnoreCase);
var staleCutoff = DateTime.UtcNow.AddDays(-Math.Max(0, refreshOlderThanDays));
var toScrape = new List<string>();
foreach (var url in allUrls)
{
if (resultByUrl.TryGetValue(url, out var existing))
{
if (refreshOlderThanDays > 0 && existing.ScrapedAt < staleCutoff)
toScrape.Add(url); // stale → refresh for price changes
}
else
{
if (retryErrors || !errorUrls.Contains(url))
toScrape.Add(url); // never scraped (skip known errors unless retrying)
}
}
if (maxProducts > 0)
toScrape = toScrape.Take(maxProducts).ToList();
var total = toScrape.Count;
Log.Info($"URLs: {allUrls.Count}; already scraped: {resultByUrl.Count}; errors on file: {errorUrls.Count}");
Log.Info($"To scrape this run: {total} (refresh older than {refreshOlderThanDays}d, retry errors: {retryErrors})");
if (total == 0)
{
Log.Info("Nothing to scrape. Done.");
return (0, 0);
}
var avgDelaySec = (_config.MinDelaySeconds + _config.MaxDelaySeconds) / 2.0;
var etaMinutes = total * (avgDelaySec + _config.PageSettleSeconds + 2) / 60.0;
Log.Info($"Estimated run time: ~{FormatDuration(TimeSpan.FromMinutes(etaMinutes))} " +
$"(grab a coffee if that's a while — it saves after every product and is resumable).");
var stopwatch = Stopwatch.StartNew();
int scraped = 0, errors = 0, index = 0, consecutiveBlocks = 0;
foreach (var url in toScrape)
{
index++;
for (var attempt = 1; ; attempt++)
{
try
{
var row = await ParseProductAsync(url, index, total);
if (resultByUrl.TryGetValue(url, out var existing))
data.Results[data.Results.IndexOf(existing)] = row;
else
data.Results.Add(row);
resultByUrl[url] = row;
data.Errors.RemoveAll(e => JsonStore.CleanUrl(e.ProductUrl).Equals(url, StringComparison.OrdinalIgnoreCase));
scraped++;
consecutiveBlocks = 0;
JsonStore.SaveOutput(_config.OutputJsonFile, data);
var basePrice = row.PriceTiers.Count > 0 ? row.PriceTiers.Min(t => t.Price) : 0m;
Log.Info($"[{index}/{total}] Saved {row.Sku} \"{row.ColorName}\" " +
$"({row.PriceTiers.Count} tier(s), base ${basePrice:0.00}) | elapsed {FormatDuration(stopwatch.Elapsed)}");
break;
}
catch (Exception ex) when (IsBlocked(ex) && attempt <= _config.BlockedMaxRetries)
{
// Site pushed back — back off (escalating) and retry the SAME product rather
// than barreling on, which is how an unattended run gets hard-banned.
consecutiveBlocks++;
var cooldown = Math.Min(_config.BlockedCooldownSeconds * consecutiveBlocks, _config.BlockedCooldownMaxSeconds);
Log.Warn($"[{index}/{total}] Blocked (403), attempt {attempt}. Cooling down {cooldown}s, then retrying this product...");
await Task.Delay(cooldown * 1000);
}
catch (Exception ex)
{
data.Errors.Add(new ScrapeError { ProductUrl = url, Error = ex.Message, ScrapedAt = DateTime.UtcNow });
JsonStore.SaveOutput(_config.OutputJsonFile, data);
errors++;
Log.Error($"[{index}/{total}] {url} -> {ex.Message}");
break;
}
}
// Periodic longer rest — eases server load and avoids a robotic, evenly-spaced cadence.
if (_config.LongRestEveryProducts > 0 && index % _config.LongRestEveryProducts == 0 && index < total)
{
Log.Info($"Resting {_config.LongRestSeconds}s after {index} products...");
await Task.Delay(_config.LongRestSeconds * 1000);
}
if (index < total)
{
var delayMs = RandomDelayMs();
Log.Info($"[{index}/{total}] Waiting {delayMs / 1000.0:0.0}s before next product...");
await Task.Delay(delayMs);
}
}
Log.Info($"Scrape complete. Scraped {scraped}, errors {errors}. Total results on file: {data.Results.Count}. " +
$"Took {FormatDuration(stopwatch.Elapsed)}.");
return (scraped, errors);
}
private async Task<ProductRecord> ParseProductAsync(string url, int index, int total)
{
Log.Info($"[{index}/{total}] Scraping {url}");
var response = await _session.Page.GotoAsync(url, new PageGotoOptions
{
WaitUntil = WaitUntilState.DOMContentLoaded,
Timeout = 60000
});
await _session.Page.WaitForTimeoutAsync(_config.PageSettleSeconds * 1000);
var status = response?.Status ?? 0;
var title = Clean(await SafeTextAsync(() => _session.Page.TitleAsync()));
var plainText = Clean(await SafeTextAsync(() => _session.Page.Locator("body").InnerTextAsync()));
if (status == 403 || Regex.IsMatch(title, @"^403 Forbidden$", RegexOptions.IgnoreCase))
throw new Exception("403 Forbidden returned by site.");
if (status == 404 || Regex.IsMatch(title, @"404|Page Not Found", RegexOptions.IgnoreCase))
throw new Exception("404 Not Found returned by site.");
var colorName = Clean(await SafeTextAsync(() => _session.Page.Locator("h1").First.InnerTextAsync()));
var skuMatch = SkuRegex.Match(plainText);
var sku = skuMatch.Success ? skuMatch.Groups[1].Value : "";
if (string.IsNullOrEmpty(sku) && string.IsNullOrEmpty(colorName))
throw new Exception("Could not find SKU or title on product page.");
var descMatch = DescRegex.Match(plainText);
var description = descMatch.Success ? Clean(descMatch.Groups[1].Value) : "";
return new ProductRecord
{
Sku = sku,
ColorName = colorName,
Description = description,
PriceTiers = ParsePriceTiers(plainText),
SafetyDataSheetUrl = await GetLinkByTextAsync(new[] { "Safety Data Sheet", @"\bSDS\b" }),
TechnicalDataSheetUrl = await GetLinkByTextAsync(new[] { "Tech Data Sheet", "Technical Data Sheet", @"\bTDS\b" }),
ApplicationGuideUrl = await GetLinkByTextAsync(new[] { "Application Guide" }),
SampleImageUrl = await GetSampleImageUrlAsync(),
ProductUrl = url,
ScrapedAt = DateTime.UtcNow
};
}
private static List<PriceTier> ParsePriceTiers(string text)
{
var tiers = new List<PriceTier>();
foreach (Match m in PriceTierRegex.Matches(text))
{
if (!decimal.TryParse(m.Groups[2].Value, NumberStyles.Any, CultureInfo.InvariantCulture, out var price))
continue;
var rangeText = Clean(m.Groups[1].Value);
int? min = null, max = null;
var range = RangeRegex.Match(rangeText);
if (range.Success)
{
min = int.Parse(range.Groups[1].Value);
max = int.Parse(range.Groups[2].Value);
}
var plus = PlusRegex.Match(rangeText);
if (plus.Success)
{
min = int.Parse(plus.Groups[1].Value);
max = null;
}
tiers.Add(new PriceTier { Min = min, Max = max, Price = price });
}
return tiers;
}
/// <summary>Returns the href of the first link whose text matches any pattern. Uses a single eval
/// returning "texthref" pairs to avoid object deserialization quirks.</summary>
private async Task<string> GetLinkByTextAsync(string[] patterns)
{
var combined = await _session.Page.EvalOnSelectorAllAsync<string[]>(
"a",
"els => els.map(a => ((a.innerText || a.textContent || '').replace(/\\s+/g, ' ').trim()) " +
"+ String.fromCharCode(1) + (a.href || ''))");
foreach (var entry in combined)
{
var parts = entry.Split('');
var text = parts.Length > 0 ? parts[0] : "";
var href = parts.Length > 1 ? parts[1] : "";
// Require the link to point at an actual document, not a generic /documents nav page.
if (href.Length > 0
&& IsDocumentUrl(href)
&& patterns.Any(p => Regex.IsMatch(text, p, RegexOptions.IgnoreCase)))
return href;
}
return "";
}
/// <summary>True when an href looks like a real document (hosted on the NIC CDN or a direct PDF).</summary>
private static bool IsDocumentUrl(string href)
{
var path = href.Split('?')[0];
return href.Contains("nicindustries.com", StringComparison.OrdinalIgnoreCase)
|| path.EndsWith(".pdf", StringComparison.OrdinalIgnoreCase);
}
private async Task<string> GetSampleImageUrlAsync()
{
var srcs = await _session.Page.EvalOnSelectorAllAsync<string[]>(
"img",
"els => els.map(i => i.currentSrc || i.src || i.getAttribute('src') || i.getAttribute('data-src') || '')" +
".filter(Boolean)");
// Only accept real product images on the NIC CDN (prefer full-size over thumbnail). Do NOT
// fall back to any "prismatic"-ish URL — that catches the site logo on products with no image.
return srcs.FirstOrDefault(s => Regex.IsMatch(s, @"images\.nicindustries\.com/prismatic/products", RegexOptions.IgnoreCase)
&& !Regex.IsMatch(s, "thumbnail", RegexOptions.IgnoreCase))
?? srcs.FirstOrDefault(s => Regex.IsMatch(s, @"images\.nicindustries\.com/prismatic/products", RegexOptions.IgnoreCase))
?? "";
}
private static bool IsBlocked(Exception ex) =>
ex.Message.Contains("403", StringComparison.OrdinalIgnoreCase);
private static async Task<string> SafeTextAsync(Func<Task<string>> fn)
{
try { return await fn(); } catch { return ""; }
}
private static string Clean(string? text) => WhitespaceRegex.Replace(text ?? "", " ").Trim();
private int RandomDelayMs()
{
var min = Math.Max(0, _config.MinDelaySeconds * 1000);
var max = Math.Max(min, _config.MaxDelaySeconds * 1000);
return _random.Next(min, max + 1);
}
private static string FormatDuration(TimeSpan t) =>
t.TotalHours >= 1 ? $"{(int)t.TotalHours}h {t.Minutes}m" :
t.TotalMinutes >= 1 ? $"{(int)t.TotalMinutes}m {t.Seconds}s" :
$"{t.Seconds}s";
}
@@ -0,0 +1,38 @@
{
"Sync": {
"BaseUrl": "https://www.prismaticpowders.com",
"ColorsPath": "/shop/powder-coating-colors",
"ProductUrlsFile": "product-urls.txt",
"OutputJsonFile": "prismatic_powders.json",
"LogFile": "prismatic-sync.log",
"MinDelaySeconds": 6,
"MaxDelaySeconds": 14,
"PageSettleSeconds": 4,
"BlockedCooldownSeconds": 120,
"BlockedCooldownMaxSeconds": 600,
"BlockedMaxRetries": 3,
"LongRestEveryProducts": 150,
"LongRestSeconds": 45,
"ScrollWaitMs": 1500,
"MaxScrolls": 400,
"StopAfterNoNewScrolls": 10,
"StopAfterKnownScrolls": 8,
"ColorParams": [
"pris_black", "pris_blue", "pris_bronze", "pris_brown", "pris_clear",
"pris_copper", "pris_gold", "pris_gray", "pris_green", "pris_orange",
"pris_pink", "pris_purple", "pris_red", "pris_silver", "pris_tan",
"pris_white", "pris_yellow"
],
"Import": {
"EndpointUrl": "",
"Token": "",
"VendorName": "Prismatic Powders"
}
}
}
@@ -0,0 +1,46 @@
namespace PowderCoating.Application.Constants;
/// <summary>
/// Central constants for the Columbia Coatings catalog integration — config keys, platform-setting
/// keys, and the derived provenance/manufacturer/category values the mapper assigns. Kept in one
/// place so the API client, sync mapper, scheduled job, and purge logic all agree on the strings.
/// </summary>
public static class ColumbiaIntegrationConstants
{
// ── Configuration keys (appsettings.json / environment) ───────────────
/// <summary>API key — secret, lives in config not the platform-settings DB.</summary>
public const string ConfigApiKey = "Columbia:ApiKey";
public const string ConfigBaseUrl = "Columbia:BaseUrl";
/// <summary>Configurable API namespace/base path, so an API version bump is a config change.</summary>
public const string ConfigApiBasePath = "Columbia:ApiBasePath";
public const string DefaultBaseUrl = "https://columbiacoatings.com";
public const string DefaultApiBasePath = "/wp-json/cca/v1";
/// <summary>Resource segment appended to the API base path for product endpoints.</summary>
public const string ProductsResource = "/products";
/// <summary>API caps per_page at 100.</summary>
public const int MaxPerPage = 100;
// ── Platform setting keys (SuperAdmin-managed, non-secret) ────────────
public const string SettingEnabled = "ColumbiaSyncEnabled";
public const string SettingIntervalDays = "ColumbiaSyncIntervalDays";
public const string SettingLastSyncedAt = "ColumbiaLastSyncedAt";
public const string SettingLastResult = "ColumbiaLastSyncResult";
public const int DefaultSyncIntervalDays = 7;
// ── Provenance ────────────────────────────────────────────────────────
/// <summary>Stored in <c>PowderCatalogItem.Source</c> — the purge key for right-to-delete.</summary>
public const string SourceName = "Columbia Coatings API";
// ── Derived manufacturers (PowderCatalogItem.VendorName) ──────────────
public const string ManufacturerColumbia = "Columbia Coatings";
public const string ManufacturerPpg = "PPG";
public const string ManufacturerKp = "KP Pigments";
// ── Derived category (PowderCatalogItem.Category) ─────────────────────
public const string CategoryPowderAdditives = "Powder Additives";
}
@@ -0,0 +1,49 @@
using System.Text.Json;
using System.Text.Json.Serialization;
namespace PowderCoating.Application.DTOs.Columbia;
/// <summary>
/// Tolerant converter for Columbia image fields. WordPress/WooCommerce returns an object
/// (<c>{id,src,name,alt}</c>) when an image is present, but an empty array (<c>[]</c>) — or
/// sometimes <c>false</c>/empty string — when it is not. A single <see cref="ColumbiaImage"/>
/// can't bind to those non-object forms, so this converter reads the object when present and
/// yields null for anything else (consuming the token so deserialization continues).
/// </summary>
public class ColumbiaImageJsonConverter : JsonConverter<ColumbiaImage?>
{
public override ColumbiaImage? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
{
switch (reader.TokenType)
{
case JsonTokenType.StartObject:
using (var doc = JsonDocument.ParseValue(ref reader))
{
var el = doc.RootElement;
return new ColumbiaImage
{
Id = el.TryGetProperty("id", out var id) && id.TryGetInt32(out var i) ? i : 0,
Src = GetString(el, "src"),
Name = GetString(el, "name"),
Alt = GetString(el, "alt"),
};
}
case JsonTokenType.StartArray:
reader.Skip(); // empty/non-empty array form means "no image"
return null;
default:
// Primitive (false / "" / null / number): nothing to consume further.
return null;
}
}
public override void Write(Utf8JsonWriter writer, ColumbiaImage? value, JsonSerializerOptions options)
=> throw new NotSupportedException("Columbia image fields are read-only.");
private static string GetString(JsonElement el, string property) =>
el.TryGetProperty(property, out var v) && v.ValueKind == JsonValueKind.String
? v.GetString() ?? string.Empty
: string.Empty;
}
@@ -0,0 +1,142 @@
using System.Text.Json;
using System.Text.Json.Serialization;
namespace PowderCoating.Application.DTOs.Columbia;
/// <summary>
/// One page of the Columbia Coatings <c>GET /products</c> response: a list of products plus
/// pagination metadata. Property names are snake_case in the API and bound via the snake-case
/// naming policy configured on the client's <see cref="JsonSerializerOptions"/>.
/// </summary>
public class ColumbiaProductsResponse
{
public List<ColumbiaProduct> Items { get; set; } = new();
public ColumbiaPagination? Pagination { get; set; }
}
/// <summary>Pagination block returned alongside a product page.</summary>
public class ColumbiaPagination
{
public int Page { get; set; }
public int PerPage { get; set; }
public int Total { get; set; }
public int TotalPages { get; set; }
}
/// <summary>
/// A single Columbia Coatings product as returned by the API. This mirrors the wire shape, not our
/// catalog model — mapping into <c>PowderCatalogItem</c> happens in the sync mapper. Prices arrive
/// as strings; cure/spec fields are free text; documents are direct URLs.
/// </summary>
public class ColumbiaProduct
{
public int Id { get; set; }
/// <summary>"simple" or "variable". Variable products carry packaging/size variants in
/// <see cref="VariationPricing"/> and leave <see cref="TieredPricing"/> as an empty array.</summary>
public string ProductType { get; set; } = string.Empty;
public string Name { get; set; } = string.Empty;
public string Slug { get; set; } = string.Empty;
public string Sku { get; set; } = string.Empty;
public string Permalink { get; set; } = string.Empty;
public string Status { get; set; } = string.Empty;
/// <summary>Columbia-specific values seen include "In Stock"/"instock", "formulated",
/// "drop_shipped", "multiple_variations", "outofstock", "onbackorder" — mixed casing.</summary>
public string StockStatus { get; set; } = string.Empty;
// ── Pricing (all strings on the wire) ─────────────────────────────────
public string Price { get; set; } = string.Empty;
public string RegularPrice { get; set; } = string.Empty;
public string SalePrice { get; set; } = string.Empty;
/// <summary>
/// Quantity-break pricing. POLYMORPHIC on the wire: an object
/// (<c>{type, minimum_quantity, tiers:[...]}</c>) on simple products, but an empty ARRAY
/// (<c>[]</c>) on variable products. Captured as a nullable raw <see cref="JsonElement"/> so
/// deserialization never throws on the type mismatch and an absent value is null (not an
/// invalid <c>Undefined</c> element); the mapper interprets it.
/// </summary>
public JsonElement? TieredPricing { get; set; }
/// <summary>Per-variant pricing for variable products (e.g. Bulk vs 1 lb Bags, or gram sizes).
/// Each variant has its own SKU and price. Empty for simple products.</summary>
public List<ColumbiaVariationPricing>? VariationPricing { get; set; }
// ── Documents (direct URLs) ───────────────────────────────────────────
public string SafetyDataSheet { get; set; } = string.Empty;
public string TechnicalDataSheet { get; set; } = string.Empty;
public string ProductFlyer { get; set; } = string.Empty;
public string ProductBrochure { get; set; } = string.Empty;
// ── Coating spec free-text ────────────────────────────────────────────
public string CureSchedule { get; set; } = string.Empty;
public string MilThickness { get; set; } = string.Empty;
/// <summary>Resin chemistry (e.g. "Polyester/TGIC", "TGIC", "Epoxy"). NOT finish/gloss.</summary>
public string Type { get; set; } = string.Empty;
public string ReleaseDate { get; set; } = string.Empty;
public string FormulationDate { get; set; } = string.Empty;
/// <summary>Free-text reformulation log, e.g. "Formulation Change: 05/22/26".</summary>
public string FormulationDateChanges { get; set; } = string.Empty;
// ── Content ───────────────────────────────────────────────────────────
/// <summary>HTML product description (WordPress markup).</summary>
public string Description { get; set; } = string.Empty;
public string ShortDescription { get; set; } = string.Empty;
public ColumbiaImage? FeaturedImage { get; set; }
// ── Taxonomy (arrays of {name}) — used at import to derive manufacturer/category, not stored raw ──
public List<ColumbiaNamed> Categories { get; set; } = new();
public List<ColumbiaNamed> Tags { get; set; } = new();
public List<ColumbiaNamed> PaColorGroup { get; set; } = new();
public List<ColumbiaAttribute> Attributes { get; set; } = new();
}
/// <summary>A pricing variant of a variable product (own SKU, own price/tiers).</summary>
public class ColumbiaVariationPricing
{
public int Id { get; set; }
public string Sku { get; set; } = string.Empty;
public List<ColumbiaAttributeValue> Attributes { get; set; } = new();
public string StockStatus { get; set; } = string.Empty;
public string Price { get; set; } = string.Empty;
public string RegularPrice { get; set; } = string.Empty;
public string SalePrice { get; set; } = string.Empty;
/// <summary>Same polymorphic object-or-array shape as the parent; captured raw (nullable).</summary>
public JsonElement? TieredPricing { get; set; }
}
/// <summary>An image object — featured or gallery.</summary>
public class ColumbiaImage
{
public int Id { get; set; }
public string Src { get; set; } = string.Empty;
public string Name { get; set; } = string.Empty;
public string Alt { get; set; } = string.Empty;
}
/// <summary>A simple <c>{name}</c> taxonomy entry (category, tag, or color group).</summary>
public class ColumbiaNamed
{
public string Name { get; set; } = string.Empty;
}
/// <summary>A product attribute with its option list, e.g. Color Group → [Blue], Packaging → [Bulk, 1 lb Bags].</summary>
public class ColumbiaAttribute
{
public string Name { get; set; } = string.Empty;
public List<ColumbiaNamed> Options { get; set; } = new();
}
/// <summary>A resolved attribute value on a specific variation, e.g. Packaging → "Bulk".</summary>
public class ColumbiaAttributeValue
{
public string Name { get; set; } = string.Empty;
public string Value { get; set; } = string.Empty;
}
@@ -37,6 +37,8 @@ public class InventoryItemDto
public decimal AverageCost { get; set; }
public decimal LastPurchasePrice { get; set; }
public DateTime? LastPurchaseDate { get; set; }
public decimal? CatalogReferencePrice { get; set; }
public DateTime? CatalogPriceUpdatedAt { get; set; }
public int? PrimaryVendorId { get; set; }
public string? PrimaryVendorName { get; set; }
public string? VendorPartNumber { get; set; }
@@ -0,0 +1,41 @@
namespace PowderCoating.Application.Interfaces;
/// <summary>
/// Orchestrates a full Columbia Coatings catalog sync: pull every product, map and upsert it, then
/// (only on a complete pull) reconcile discontinuations. Used by both the scheduled background job
/// and the manual "Sync now" admin action.
/// </summary>
public interface IColumbiaCatalogSyncService
{
/// <summary>
/// Runs one full sync. Assumes the caller has already decided it should run (enabled / due).
/// Returns a result describing the outcome; never throws for an expected failure (not
/// configured, partial pull, HTTP error) — those are reported on the result instead.
/// </summary>
Task<ColumbiaSyncResult> RunSyncAsync(CancellationToken cancellationToken = default);
}
/// <summary>Outcome of a Columbia catalog sync run.</summary>
public class ColumbiaSyncResult
{
public bool Success { get; set; }
public string? ErrorMessage { get; set; }
public int TotalFetched { get; set; }
public int Inserted { get; set; }
public int Updated { get; set; }
public int Unchanged { get; set; }
public int Skipped { get; set; }
public int Discontinued { get; set; }
public int Reactivated { get; set; }
public DateTime StartedAt { get; set; }
public TimeSpan Duration { get; set; }
/// <summary>One-line summary suitable for storing in the last-result platform setting / UI.</summary>
public string Summary =>
Success
? $"{TotalFetched} fetched: {Inserted} new, {Updated} updated, {Unchanged} unchanged, " +
$"{Discontinued} discontinued, {Reactivated} reactivated ({Duration.TotalSeconds:F0}s)"
: $"Failed: {ErrorMessage}";
}
@@ -0,0 +1,47 @@
using PowderCoating.Application.DTOs.Columbia;
namespace PowderCoating.Application.Interfaces;
/// <summary>
/// Typed client for the Columbia Coatings product catalog API (<c>/wp-json/cca/v1</c>).
/// Read-only: lists products via the paged <c>GET /products</c> endpoint.
/// <para>
/// We deliberately page <c>/products</c> rather than using the bulk <c>export.json</c> download:
/// the export returns a temporary <c>download_url</c> to a static file under <c>/wp-content/uploads</c>,
/// which sits behind Cloudflare bot protection and 403s for non-browser clients. The
/// <c>/wp-json</c> API routes are allowlisted via the API key, so paging is the only path that
/// works reliably from a server.
/// </para>
/// </summary>
public interface IColumbiaCoatingsApiClient
{
/// <summary>
/// True when an API key is configured (<c>Columbia:ApiKey</c>). When false, callers should
/// skip the sync entirely rather than issue unauthenticated requests.
/// </summary>
bool IsConfigured { get; }
/// <summary>
/// Retrieves a single page of products. <paramref name="perPage"/> is capped at 100 by the API.
/// </summary>
Task<ColumbiaProductsResponse> GetProductsPageAsync(int page, int perPage, CancellationToken cancellationToken = default);
/// <summary>
/// Pages through the entire catalog and returns every product. Honors rate limiting
/// (429 / Retry-After). THROWS if any page fails after retries — callers must treat an
/// exception as "incomplete pull" and NOT run discontinuation logic against a partial set.
/// </summary>
Task<IReadOnlyList<ColumbiaProduct>> GetAllProductsAsync(CancellationToken cancellationToken = default);
/// <summary>
/// Fetches a single product by exact SKU (<c>GET /products?sku=...</c>), or null if not found.
/// For ad-hoc refresh of one record without pulling the whole catalog.
/// </summary>
Task<ColumbiaProduct?> GetProductBySkuAsync(string sku, CancellationToken cancellationToken = default);
/// <summary>
/// Fetches a single product by WooCommerce product ID (<c>GET /products/{id}</c>), or null if
/// not found. Useful when we already store the catalog product's ID and want to refresh it.
/// </summary>
Task<ColumbiaProduct?> GetProductByIdAsync(int id, CancellationToken cancellationToken = default);
}
@@ -59,9 +59,17 @@ public interface IInventoryAiLookupService
Task<InventoryAiLookupResult> ScanLabelAsync(string base64Image, string mediaType);
/// <summary>
/// Fetches a Technical Data Sheet URL and extracts cure temperature and cure time.
/// Called when the main lookup found a TDS URL but cure specs are still missing.
/// Fetches a Technical Data Sheet URL and extracts cure temperature, cure time, and specific
/// gravity. Called when the main lookup found a TDS URL but specs are still missing.
/// Returns Success=false silently (no UI error) when the TDS is a PDF or unreachable.
/// </summary>
Task<InventoryAiLookupResult> FetchTdsCureSpecsAsync(string tdsUrl, string? colorName);
/// <summary>
/// Lazily fills a powder catalog item's specific gravity (and any missing cure specs) from its
/// TDS the first time it's needed, then derives theoretical coverage. No-op when specific
/// gravity is already known or no TDS URL is present. Persists the enrichment to the catalog so
/// it's done once and benefits every future use. Returns true if anything was filled.
/// </summary>
Task<bool> EnsureCatalogTdsSpecsAsync(PowderCoating.Core.Entities.PowderCatalogItem catalog);
}
@@ -0,0 +1,32 @@
using PowderCoating.Core.Entities;
namespace PowderCoating.Application.Interfaces;
/// <summary>
/// Shared upsert for the platform powder catalog: matches incoming records to existing rows by
/// (VendorName, SKU), inserts new ones, and updates changed ones in place. Used by BOTH the manual
/// JSON file import and the Columbia API sync so there is a single upsert path, only the mapping
/// differs. Does NOT handle discontinuation — that is a sync-specific concern.
/// </summary>
public interface IPowderCatalogUpsertService
{
/// <summary>
/// Applies <paramref name="incoming"/> mapped catalog items. Only fields sourced from the feed
/// are copied on update; enrichment fields (specific gravity, coverage, transfer efficiency,
/// finish) are preserved so they are not wiped by a feed that never carries them. Changed and
/// inserted rows get <paramref name="runTimestamp"/> stamped on LastSyncedAt/UpdatedAt.
/// </summary>
Task<PowderCatalogUpsertResult> UpsertAsync(
IReadOnlyList<PowderCatalogItem> incoming,
DateTime runTimestamp,
CancellationToken cancellationToken = default);
}
/// <summary>Counts from an upsert run.</summary>
public class PowderCatalogUpsertResult
{
public int Inserted { get; set; }
public int Updated { get; set; }
public int Unchanged { get; set; }
public int Skipped { get; set; }
}
@@ -149,9 +149,12 @@ public class PricingCalculationService : IPricingCalculationService
try
{
var inventoryItem = await _unitOfWork.InventoryItems.GetByIdAsync(coat.InventoryItemId.Value);
if (inventoryItem != null && inventoryItem.UnitCost > 0)
// Prefer the current catalog price (replacement cost) so quotes reflect the latest
// price; fall back to the item's own cost when it isn't catalog-linked.
var effectiveCostPerLb = inventoryItem?.CatalogReferencePrice ?? inventoryItem?.UnitCost ?? 0m;
if (inventoryItem != null && effectiveCostPerLb > 0)
{
costPerLb = inventoryItem.UnitCost;
costPerLb = effectiveCostPerLb;
isIncomingPowder = inventoryItem.IsIncoming;
var coverage = coat.CoverageSqFtPerLb;
var transferEfficiency = coat.TransferEfficiency;
@@ -160,8 +163,8 @@ public class PricingCalculationService : IPricingCalculationService
var actualPoundsPerSqFt = poundsPerSqFt / (transferEfficiency / 100m);
powderCostPerSqFt = actualPoundsPerSqFt * costPerLb;
_logger.LogInformation("Coat {CoatName}: Using inventory item: {InventoryItem} (IsIncoming={IsIncoming}), UnitCost={UnitCost}/lb, Coverage={Coverage}sqft/lb, Efficiency={Efficiency}%, Calculated={CalcCost}/sqft",
coat.CoatName, inventoryItem.Name, isIncomingPowder, inventoryItem.UnitCost, coverage, transferEfficiency, powderCostPerSqFt);
_logger.LogInformation("Coat {CoatName}: Using inventory item: {InventoryItem} (IsIncoming={IsIncoming}), CostPerLb={CostPerLb}/lb (catalog ref={CatalogRef}), Coverage={Coverage}sqft/lb, Efficiency={Efficiency}%, Calculated={CalcCost}/sqft",
coat.CoatName, inventoryItem.Name, isIncomingPowder, costPerLb, inventoryItem.CatalogReferencePrice, coverage, transferEfficiency, powderCostPerSqFt);
}
}
catch (Exception ex)
@@ -691,7 +694,8 @@ public class PricingCalculationService : IPricingCalculationService
var invItem = await _unitOfWork.InventoryItems.GetByIdAsync(c.InventoryItemId.Value);
if (invItem?.IsIncoming == true)
{
customPowderOrderAmount += c.PowderToOrder.Value * invItem.UnitCost;
// Bill the powder-to-order at the current catalog price when linked.
customPowderOrderAmount += c.PowderToOrder.Value * (invItem.CatalogReferencePrice ?? invItem.UnitCost);
var colorName = !string.IsNullOrWhiteSpace(c.ColorName) ? c.ColorName : invItem.Name;
if (!string.IsNullOrWhiteSpace(colorName))
customPowderOrderColors.Add(colorName);
@@ -31,6 +31,27 @@ public class InventoryItem : BaseEntity
public string? SdsUrl { get; set; } // Safety Data Sheet URL (from powder catalog or manual entry)
public string? TdsUrl { get; set; } // Technical Data Sheet URL (from powder catalog or manual entry)
/// <summary>
/// Optional link to the platform powder catalog record this item was created from.
/// Populated when an item is added via the catalog lookup, or back-filled by Manufacturer+SKU.
/// Lets the inventory detail screen surface manufacturer-level status (e.g. "discontinued by
/// manufacturer — cannot reorder") and future price/reformulation change flags. Nulled — not
/// cascaded — if the source catalog data is purged (the shop's own stock record must survive).
/// </summary>
public int? PowderCatalogItemId { get; set; }
/// <summary>
/// Latest list price from the linked powder catalog, refreshed by the catalog sync. This is the
/// QUOTING price (current replacement cost) and is kept deliberately SEPARATE from
/// <see cref="UnitCost"/>/<see cref="AverageCost"/> (the actual paid cost basis that drives
/// inventory valuation and COGS). Quoting prefers this when present so quotes reflect the
/// current price; accounting never reads it. Null for manual/non-catalog powders.
/// </summary>
public decimal? CatalogReferencePrice { get; set; }
/// <summary>Timestamp (UTC) when <see cref="CatalogReferencePrice"/> was last refreshed by the sync.</summary>
public DateTime? CatalogPriceUpdatedAt { get; set; }
// Sample Panel Tracking (coating category items only)
public bool HasSamplePanel { get; set; } = false;
@@ -40,9 +40,30 @@ public class PowderCatalogItem
/// <summary>Cure hold time at cure temperature, in minutes.</summary>
public int? CureTimeMinutes { get; set; }
/// <summary>
/// Raw cure schedule text exactly as supplied by the vendor — e.g. "10 minutes @ 400°F".
/// Preserved verbatim because vendor formats vary wildly and some carry application notes
/// that don't reduce to a single temp/time pair (partial cures, clear-coat steps).
/// </summary>
public string? CureScheduleText { get; set; }
/// <summary>
/// All parsed cure curves as JSON — e.g. [{"tempF":400,"minutes":10},{"tempF":350,"minutes":20}].
/// Many powders list alternate lower-temperature curves; these matter for heat-sensitive
/// substrates that cannot take the standard 400°F cure, so we keep every curve, not just the
/// primary one in <see cref="CureTemperatureF"/>/<see cref="CureTimeMinutes"/>.
/// </summary>
public string? CureCurvesJson { get; set; }
/// <summary>Finish type — e.g. Gloss, Matte, Satin, Metallic, Texture.</summary>
public string? Finish { get; set; }
/// <summary>Resin chemistry — e.g. "Polyester", "TGIC", "Epoxy", "Hybrid". Distinct from <see cref="Finish"/>.</summary>
public string? ChemistryType { get; set; }
/// <summary>Recommended film build (mil thickness) as free text from the vendor — e.g. "2.0-3.0 Mils".</summary>
public string? MilThickness { get; set; }
/// <summary>Comma-separated color family tags — e.g. "Blue,Purple".</summary>
public string? ColorFamilies { get; set; }
@@ -60,6 +81,29 @@ public class PowderCatalogItem
// ── Catalog management ────────────────────────────────────────────────
/// <summary>
/// Our internal product category — e.g. "Powder Additives" for pigments/additives that are
/// sold by weight in grams and mixed into clear rather than sprayed as a standalone powder.
/// Null/empty for standard powders. Derived at import from the vendor's taxonomy, NOT stored
/// from their raw category list.
/// </summary>
public string? Category { get; set; }
/// <summary>
/// Provenance of this record — e.g. "Columbia Coatings API". Kept SEPARATE from
/// <see cref="VendorName"/> (which holds the derived manufacturer) so we can honor a
/// distributor's right-to-delete by purging every record that came from their feed,
/// regardless of which manufacturer made the product.
/// </summary>
public string? Source { get; set; }
/// <summary>
/// Reformulation history as supplied by the vendor — e.g. "Formulation Change: 05/22/26".
/// Not a reliable modified-date (free text, reformulations only) but a useful signal that a
/// product's formula — and therefore its cure specs — may have changed.
/// </summary>
public string? FormulationChanges { get; set; }
/// <summary>True when the vendor has discontinued this product. Kept for historical lookups.</summary>
public bool IsDiscontinued { get; set; } = false;
@@ -1511,6 +1511,9 @@ modelBuilder.Entity<Job>()
modelBuilder.Entity<InventoryItem>()
.HasIndex(i => new { i.CompanyId, i.SKU })
.IsUnique()
// Filter on IsDeleted so soft-deleted items don't reserve their SKU and block a new
// (or re-created) item from reusing it — matching the app's soft-delete semantics.
.HasFilter("[IsDeleted] = 0")
.HasDatabaseName("IX_InventoryItems_CompanyId_SKU");
modelBuilder.Entity<Company>()
@@ -0,0 +1,141 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace PowderCoating.Infrastructure.Migrations
{
/// <inheritdoc />
public partial class AddColumbiaCatalogIntegrationFields : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<string>(
name: "Category",
table: "PowderCatalogItems",
type: "nvarchar(max)",
nullable: true);
migrationBuilder.AddColumn<string>(
name: "ChemistryType",
table: "PowderCatalogItems",
type: "nvarchar(max)",
nullable: true);
migrationBuilder.AddColumn<string>(
name: "CureCurvesJson",
table: "PowderCatalogItems",
type: "nvarchar(max)",
nullable: true);
migrationBuilder.AddColumn<string>(
name: "CureScheduleText",
table: "PowderCatalogItems",
type: "nvarchar(max)",
nullable: true);
migrationBuilder.AddColumn<string>(
name: "FormulationChanges",
table: "PowderCatalogItems",
type: "nvarchar(max)",
nullable: true);
migrationBuilder.AddColumn<string>(
name: "MilThickness",
table: "PowderCatalogItems",
type: "nvarchar(max)",
nullable: true);
migrationBuilder.AddColumn<string>(
name: "Source",
table: "PowderCatalogItems",
type: "nvarchar(max)",
nullable: true);
migrationBuilder.AddColumn<int>(
name: "PowderCatalogItemId",
table: "InventoryItems",
type: "int",
nullable: true);
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 1,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 21, 27, 126, DateTimeKind.Utc).AddTicks(6644));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 2,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 21, 27, 126, DateTimeKind.Utc).AddTicks(6651));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 3,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 21, 27, 126, DateTimeKind.Utc).AddTicks(6652));
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropColumn(
name: "Category",
table: "PowderCatalogItems");
migrationBuilder.DropColumn(
name: "ChemistryType",
table: "PowderCatalogItems");
migrationBuilder.DropColumn(
name: "CureCurvesJson",
table: "PowderCatalogItems");
migrationBuilder.DropColumn(
name: "CureScheduleText",
table: "PowderCatalogItems");
migrationBuilder.DropColumn(
name: "FormulationChanges",
table: "PowderCatalogItems");
migrationBuilder.DropColumn(
name: "MilThickness",
table: "PowderCatalogItems");
migrationBuilder.DropColumn(
name: "Source",
table: "PowderCatalogItems");
migrationBuilder.DropColumn(
name: "PowderCatalogItemId",
table: "InventoryItems");
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 1,
column: "CreatedAt",
value: new DateTime(2026, 6, 14, 1, 21, 46, 131, DateTimeKind.Utc).AddTicks(4191));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 2,
column: "CreatedAt",
value: new DateTime(2026, 6, 14, 1, 21, 46, 131, DateTimeKind.Utc).AddTicks(4196));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 3,
column: "CreatedAt",
value: new DateTime(2026, 6, 14, 1, 21, 46, 131, DateTimeKind.Utc).AddTicks(4197));
}
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,88 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace PowderCoating.Infrastructure.Migrations
{
/// <inheritdoc />
public partial class SeedColumbiaSyncSettings : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
// Seed the SuperAdmin-managed platform settings for the Columbia Coatings catalog sync.
// Idempotent so it is safe against a DB where keys were added manually. The API key
// itself is NOT here — secrets live in configuration (Columbia:ApiKey), not this table.
migrationBuilder.Sql(@"
IF NOT EXISTS (SELECT 1 FROM PlatformSettings WHERE [Key] = 'ColumbiaSyncEnabled')
INSERT INTO PlatformSettings ([Key],[Value],[Label],[Description],[GroupName])
VALUES ('ColumbiaSyncEnabled','false','Columbia Coatings Sync Enabled','Master switch for the scheduled Columbia Coatings catalog sync. When off, no automatic or manual sync runs regardless of the configured API key.','Integrations');
IF NOT EXISTS (SELECT 1 FROM PlatformSettings WHERE [Key] = 'ColumbiaSyncIntervalDays')
INSERT INTO PlatformSettings ([Key],[Value],[Label],[Description],[GroupName])
VALUES ('ColumbiaSyncIntervalDays','7','Columbia Sync Interval (days)','How many days between automatic Columbia catalog syncs. A full sync is cheap (~25 API calls), so daily (1) or weekly (7) keeps pricing fresh.','Integrations');
IF NOT EXISTS (SELECT 1 FROM PlatformSettings WHERE [Key] = 'ColumbiaLastSyncedAt')
INSERT INTO PlatformSettings ([Key],[Value],[Label],[Description],[GroupName])
VALUES ('ColumbiaLastSyncedAt',NULL,'Columbia Last Synced At','Timestamp (UTC) of the last successful Columbia catalog sync. Set automatically by the sync job.','Integrations');
IF NOT EXISTS (SELECT 1 FROM PlatformSettings WHERE [Key] = 'ColumbiaLastSyncResult')
INSERT INTO PlatformSettings ([Key],[Value],[Label],[Description],[GroupName])
VALUES ('ColumbiaLastSyncResult',NULL,'Columbia Last Sync Result','Summary of the last Columbia catalog sync run (inserted/updated/discontinued counts or error). Set automatically by the sync job.','Integrations');
");
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 1,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 44, 40, 147, DateTimeKind.Utc).AddTicks(5997));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 2,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 44, 40, 147, DateTimeKind.Utc).AddTicks(6002));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 3,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 44, 40, 147, DateTimeKind.Utc).AddTicks(6003));
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.Sql(@"
DELETE FROM PlatformSettings WHERE [Key] IN (
'ColumbiaSyncEnabled','ColumbiaSyncIntervalDays','ColumbiaLastSyncedAt','ColumbiaLastSyncResult'
);
");
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 1,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 21, 27, 126, DateTimeKind.Utc).AddTicks(6644));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 2,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 21, 27, 126, DateTimeKind.Utc).AddTicks(6651));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 3,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 21, 27, 126, DateTimeKind.Utc).AddTicks(6652));
}
}
}
@@ -0,0 +1,82 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace PowderCoating.Infrastructure.Migrations
{
/// <inheritdoc />
public partial class FilterInventorySkuUniqueIndexOnSoftDelete : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropIndex(
name: "IX_InventoryItems_CompanyId_SKU",
table: "InventoryItems");
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 1,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 16, 33, 6, 662, DateTimeKind.Utc).AddTicks(4251));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 2,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 16, 33, 6, 662, DateTimeKind.Utc).AddTicks(4258));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 3,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 16, 33, 6, 662, DateTimeKind.Utc).AddTicks(4259));
migrationBuilder.CreateIndex(
name: "IX_InventoryItems_CompanyId_SKU",
table: "InventoryItems",
columns: new[] { "CompanyId", "SKU" },
unique: true,
filter: "[IsDeleted] = 0");
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropIndex(
name: "IX_InventoryItems_CompanyId_SKU",
table: "InventoryItems");
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 1,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 44, 40, 147, DateTimeKind.Utc).AddTicks(5997));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 2,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 44, 40, 147, DateTimeKind.Utc).AddTicks(6002));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 3,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 14, 44, 40, 147, DateTimeKind.Utc).AddTicks(6003));
migrationBuilder.CreateIndex(
name: "IX_InventoryItems_CompanyId_SKU",
table: "InventoryItems",
columns: new[] { "CompanyId", "SKU" },
unique: true);
}
}
}
@@ -0,0 +1,81 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace PowderCoating.Infrastructure.Migrations
{
/// <inheritdoc />
public partial class AddInventoryCatalogReferencePrice : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<DateTime>(
name: "CatalogPriceUpdatedAt",
table: "InventoryItems",
type: "datetime2",
nullable: true);
migrationBuilder.AddColumn<decimal>(
name: "CatalogReferencePrice",
table: "InventoryItems",
type: "decimal(18,2)",
nullable: true);
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 1,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 19, 26, 43, 161, DateTimeKind.Utc).AddTicks(2051));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 2,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 19, 26, 43, 161, DateTimeKind.Utc).AddTicks(2056));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 3,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 19, 26, 43, 161, DateTimeKind.Utc).AddTicks(2057));
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropColumn(
name: "CatalogPriceUpdatedAt",
table: "InventoryItems");
migrationBuilder.DropColumn(
name: "CatalogReferencePrice",
table: "InventoryItems");
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 1,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 16, 33, 6, 662, DateTimeKind.Utc).AddTicks(4251));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 2,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 16, 33, 6, 662, DateTimeKind.Utc).AddTicks(4258));
migrationBuilder.UpdateData(
table: "PricingTiers",
keyColumn: "Id",
keyValue: 3,
column: "CreatedAt",
value: new DateTime(2026, 6, 17, 16, 33, 6, 662, DateTimeKind.Utc).AddTicks(4259));
}
}
}
@@ -4075,6 +4075,12 @@ namespace PowderCoating.Infrastructure.Migrations
b.Property<decimal>("AverageCost")
.HasColumnType("decimal(18,2)");
b.Property<DateTime?>("CatalogPriceUpdatedAt")
.HasColumnType("datetime2");
b.Property<decimal?>("CatalogReferencePrice")
.HasColumnType("decimal(18,2)");
b.Property<string>("Category")
.IsRequired()
.HasColumnType("nvarchar(max)");
@@ -4173,6 +4179,9 @@ namespace PowderCoating.Infrastructure.Migrations
b.Property<string>("Notes")
.HasColumnType("nvarchar(max)");
b.Property<int?>("PowderCatalogItemId")
.HasColumnType("int");
b.Property<int?>("PrimaryVendorId")
.HasColumnType("int");
@@ -4241,7 +4250,8 @@ namespace PowderCoating.Infrastructure.Migrations
b.HasIndex("CompanyId", "SKU")
.IsUnique()
.HasDatabaseName("IX_InventoryItems_CompanyId_SKU");
.HasDatabaseName("IX_InventoryItems_CompanyId_SKU")
.HasFilter("[IsDeleted] = 0");
b.HasIndex("CompanyId", "QuantityOnHand", "ReorderPoint")
.HasDatabaseName("IX_InventoryItems_CompanyId_Quantity_Reorder");
@@ -6936,6 +6946,12 @@ namespace PowderCoating.Infrastructure.Migrations
b.Property<string>("ApplicationGuideUrl")
.HasColumnType("nvarchar(max)");
b.Property<string>("Category")
.HasColumnType("nvarchar(max)");
b.Property<string>("ChemistryType")
.HasColumnType("nvarchar(max)");
b.Property<string>("ColorFamilies")
.HasColumnType("nvarchar(max)");
@@ -6949,6 +6965,12 @@ namespace PowderCoating.Infrastructure.Migrations
b.Property<DateTime>("CreatedAt")
.HasColumnType("datetime2");
b.Property<string>("CureCurvesJson")
.HasColumnType("nvarchar(max)");
b.Property<string>("CureScheduleText")
.HasColumnType("nvarchar(max)");
b.Property<decimal?>("CureTemperatureF")
.HasColumnType("decimal(18,2)");
@@ -6961,6 +6983,9 @@ namespace PowderCoating.Infrastructure.Migrations
b.Property<string>("Finish")
.HasColumnType("nvarchar(max)");
b.Property<string>("FormulationChanges")
.HasColumnType("nvarchar(max)");
b.Property<string>("ImageUrl")
.HasColumnType("nvarchar(max)");
@@ -6973,6 +6998,9 @@ namespace PowderCoating.Infrastructure.Migrations
b.Property<DateTime?>("LastSyncedAt")
.HasColumnType("datetime2");
b.Property<string>("MilThickness")
.HasColumnType("nvarchar(max)");
b.Property<string>("PriceTiersJson")
.HasColumnType("nvarchar(max)");
@@ -6989,6 +7017,9 @@ namespace PowderCoating.Infrastructure.Migrations
.IsRequired()
.HasColumnType("nvarchar(450)");
b.Property<string>("Source")
.HasColumnType("nvarchar(max)");
b.Property<decimal?>("SpecificGravity")
.HasColumnType("decimal(18,2)");
@@ -7210,7 +7241,7 @@ namespace PowderCoating.Infrastructure.Migrations
{
Id = 1,
CompanyId = 0,
CreatedAt = new DateTime(2026, 6, 14, 1, 21, 46, 131, DateTimeKind.Utc).AddTicks(4191),
CreatedAt = new DateTime(2026, 6, 17, 19, 26, 43, 161, DateTimeKind.Utc).AddTicks(2051),
Description = "Standard pricing for regular customers",
DiscountPercent = 0m,
IsActive = true,
@@ -7221,7 +7252,7 @@ namespace PowderCoating.Infrastructure.Migrations
{
Id = 2,
CompanyId = 0,
CreatedAt = new DateTime(2026, 6, 14, 1, 21, 46, 131, DateTimeKind.Utc).AddTicks(4196),
CreatedAt = new DateTime(2026, 6, 17, 19, 26, 43, 161, DateTimeKind.Utc).AddTicks(2056),
Description = "5% discount for preferred customers",
DiscountPercent = 5m,
IsActive = true,
@@ -7232,7 +7263,7 @@ namespace PowderCoating.Infrastructure.Migrations
{
Id = 3,
CompanyId = 0,
CreatedAt = new DateTime(2026, 6, 14, 1, 21, 46, 131, DateTimeKind.Utc).AddTicks(4197),
CreatedAt = new DateTime(2026, 6, 17, 19, 26, 43, 161, DateTimeKind.Utc).AddTicks(2057),
Description = "10% discount for premium customers",
DiscountPercent = 10m,
IsActive = true,
@@ -0,0 +1,299 @@
using System.Globalization;
using System.Net;
using System.Text.Json;
using System.Text.RegularExpressions;
using PowderCoating.Application.Constants;
using PowderCoating.Application.DTOs.Columbia;
using PowderCoating.Core.Entities;
namespace PowderCoating.Infrastructure.Services.Columbia;
/// <summary>
/// Maps a Columbia Coatings API product onto our platform <see cref="PowderCatalogItem"/>.
/// Pure, static, side-effect free so the tricky bits (manufacturer derivation, the free-text
/// cure-schedule parser, HTML stripping) can be unit tested directly against captured fixtures.
/// <para>
/// Columbia is a distributor reselling multiple brands, so <see cref="PowderCatalogItem.VendorName"/>
/// holds the DERIVED manufacturer (PPG / KP Pigments / Columbia) while
/// <see cref="PowderCatalogItem.Source"/> records the feed ("Columbia Coatings API") for
/// right-to-delete purges. The vendor's own categories/tags are read here only to derive the
/// manufacturer and additive flag — they are never stored raw.
/// </para>
/// </summary>
public static class ColumbiaCatalogMapper
{
/// <summary>A single parsed cure curve — hold <see cref="Minutes"/> at <see cref="TempF"/>.</summary>
public readonly record struct CureCurve(int TempF, int Minutes);
// Resin chemistries that mean "polyester + TGIC" but arrive formatted three different ways.
private static readonly Regex PolyesterTgic =
new(@"^\s*(polyester\s*[/ ]\s*tgic|tgic\s*[/ ]\s*polyester)\s*$", RegexOptions.IgnoreCase | RegexOptions.Compiled);
// "10 minutes @ 400°F", "7 minutes at 375 F", "Metal Temperature: 10 minutes at 400°F (204°C)".
// Degree glyph is optional and may be ° (U+00B0), ˚ (U+02DA), or º (U+00BA).
private static readonly Regex CureCurveRegex =
new(@"(\d+)\s*min(?:ute)?s?\.?\s*(?:@|at)\s*(\d{2,3})\s*[°˚º]?\s*F",
RegexOptions.IgnoreCase | RegexOptions.Compiled);
private static readonly Regex HtmlTag = new("<[^>]+>", RegexOptions.Compiled);
private static readonly Regex WhitespaceRun = new(@"\s{2,}", RegexOptions.Compiled);
private static readonly JsonSerializerOptions JsonOut = new() { WriteIndented = false };
/// <summary>
/// True for products that should not be in the powder catalog as standalone colors:
/// physical swatch cards (not powder at all), and tester (4 oz) / sample (5 lb) listings that
/// are just smaller SIZES of a parent powder that already exists as its own product. Detected
/// by specific SKU suffixes (-SW / -04) and unambiguous name markers ("SWATCH", "Tester",
/// "Sample ("). The sample-size "-S" SKU suffix is intentionally NOT used on its own — the
/// "Sample (" name marker catches every sample without risking a real SKU that ends in -S.
/// </summary>
public static bool IsExcludedProduct(ColumbiaProduct p)
{
var sku = p.Sku ?? string.Empty;
if (sku.EndsWith("-SW", StringComparison.OrdinalIgnoreCase)
|| sku.EndsWith("-04", StringComparison.OrdinalIgnoreCase))
return true;
var name = p.Name ?? string.Empty;
return name.Contains("SWATCH", StringComparison.OrdinalIgnoreCase)
|| name.Contains("Tester", StringComparison.OrdinalIgnoreCase)
|| name.Contains("Sample (", StringComparison.OrdinalIgnoreCase);
}
/// <summary>Maps a Columbia product into a fully populated (unsaved) catalog item.</summary>
public static PowderCatalogItem Map(ColumbiaProduct p)
{
var curves = ParseCureCurves(p.CureSchedule);
var primary = curves.Count > 0 ? curves[0] : (CureCurve?)null;
return new PowderCatalogItem
{
VendorName = DeriveManufacturer(p),
Sku = p.Sku.Trim(),
ColorName = p.Name.Trim(),
Source = ColumbiaIntegrationConstants.SourceName,
Category = IsAdditive(p) ? ColumbiaIntegrationConstants.CategoryPowderAdditives : null,
Description = StripHtml(p.Description),
UnitPrice = ParseBasePrice(p),
PriceTiersJson = BuildPriceTiersJson(p),
ImageUrl = NullIfBlank(p.FeaturedImage?.Src),
SdsUrl = NullIfBlank(p.SafetyDataSheet),
TdsUrl = NullIfBlank(p.TechnicalDataSheet),
ApplicationGuideUrl = NullIfBlank(FirstNonBlank(p.ProductFlyer, p.ProductBrochure)),
ProductUrl = NullIfBlank(p.Permalink),
ChemistryType = NormalizeChemistry(p.Type),
MilThickness = NullIfBlank(p.MilThickness),
CureScheduleText = NullIfBlank(p.CureSchedule),
CureCurvesJson = curves.Count > 0 ? JsonSerializer.Serialize(curves, JsonOut) : null,
CureTemperatureF = primary?.TempF,
CureTimeMinutes = primary?.Minutes,
RequiresClearCoat = DetectRequiresClearCoat(p),
ColorFamilies = BuildColorFamilies(p),
FormulationChanges = NullIfBlank(p.FormulationDateChanges),
// Coverage / specific gravity / transfer efficiency are not in the API — left null for
// lazy TDS/AI enrichment on first use. IsDiscontinued is handled by the sync sweep.
};
}
// ── Manufacturer derivation ───────────────────────────────────────────
/// <summary>
/// Derives the manufacturer from the product's taxonomy/SKU. Columbia resells PPG powders and
/// KP Pigments additives through the same feed; everything else is Columbia's own brand.
/// </summary>
public static string DeriveManufacturer(ColumbiaProduct p)
{
if (IsKpPigments(p))
return ColumbiaIntegrationConstants.ManufacturerKp;
if (IsPpg(p))
return ColumbiaIntegrationConstants.ManufacturerPpg;
return ColumbiaIntegrationConstants.ManufacturerColumbia;
}
private static bool IsKpPigments(ColumbiaProduct p) =>
p.Sku.StartsWith("ADD-", StringComparison.OrdinalIgnoreCase)
|| CategoryStartsWith(p, "KP");
private static bool IsPpg(ColumbiaProduct p) =>
CategoryStartsWith(p, "PPG");
/// <summary>
/// True for pigments/additives sold by weight (grams) rather than sprayed powders. These get
/// forced into the "Powder Additives" category. Keyed off the broad Additives category and the
/// ADD- SKU prefix, not just the KP brand (there are ~98 non-KP additives).
/// </summary>
public static bool IsAdditive(ColumbiaProduct p) =>
p.Sku.StartsWith("ADD-", StringComparison.OrdinalIgnoreCase)
|| p.Categories.Any(c => c.Name.Equals("Additives", StringComparison.OrdinalIgnoreCase))
|| CategoryStartsWith(p, "KP");
private static bool CategoryStartsWith(ColumbiaProduct p, string prefix) =>
p.Categories.Any(c => c.Name.StartsWith(prefix, StringComparison.OrdinalIgnoreCase));
// ── Pricing ───────────────────────────────────────────────────────────
/// <summary>
/// Base unit price = the top-level <c>price</c> (falling back to <c>regular_price</c>). For
/// variable products the parent <c>price</c> already carries the lead variant's price, while
/// <c>regular_price</c> is often "0", so price is preferred.
/// </summary>
public static decimal ParseBasePrice(ColumbiaProduct p)
{
if (TryParseMoney(p.Price, out var price) && price > 0)
return price;
if (TryParseMoney(p.RegularPrice, out var regular) && regular > 0)
return regular;
// Variable product with a zero parent price: fall back to the lowest variant price.
var variantPrices = (p.VariationPricing ?? new List<ColumbiaVariationPricing>())
.Select(v => TryParseMoney(v.Price, out var vp) ? vp : 0m)
.Where(v => v > 0)
.ToList();
return variantPrices.Count > 0 ? variantPrices.Min() : 0m;
}
/// <summary>
/// Captures quantity-break / variant pricing as JSON for later use. For variable products this
/// is the per-variant pricing (Bulk vs 1 lb Bags, gram sizes); for simple products it's the
/// tiered_pricing object. Null when neither is present.
/// </summary>
public static string? BuildPriceTiersJson(ColumbiaProduct p)
{
if (p.VariationPricing is { Count: > 0 })
return JsonSerializer.Serialize(p.VariationPricing, JsonOut);
if (p.TieredPricing is { ValueKind: JsonValueKind.Object } tiered)
{
// Only keep it if it actually carries tiers (avoid storing empty {type,...} shells).
if (tiered.TryGetProperty("tiers", out var tiers)
&& tiers.ValueKind == JsonValueKind.Array
&& tiers.GetArrayLength() > 0)
{
return tiered.GetRawText();
}
}
return null;
}
private static bool TryParseMoney(string? s, out decimal value) =>
decimal.TryParse(s, NumberStyles.Any, CultureInfo.InvariantCulture, out value);
// ── Cure schedule parsing ─────────────────────────────────────────────
/// <summary>
/// Extracts every "N minutes at/@ TTT°F" curve from a free-text cure schedule, in document
/// order. The first is treated as the primary/standard curve; the rest are alternate (often
/// lower-temperature) curves preserved for heat-sensitive substrates. Returns an empty list for
/// schedules with no parseable temp/time pair (partial-cure / clear-coat instructions).
/// </summary>
public static List<CureCurve> ParseCureCurves(string? cureSchedule)
{
var result = new List<CureCurve>();
if (string.IsNullOrWhiteSpace(cureSchedule))
return result;
foreach (Match m in CureCurveRegex.Matches(cureSchedule))
{
if (int.TryParse(m.Groups[1].Value, out var minutes)
&& int.TryParse(m.Groups[2].Value, out var tempF)
&& tempF is >= 150 and <= 600 // sanity: real cure temps
&& minutes is > 0 and <= 120)
{
var curve = new CureCurve(tempF, minutes);
if (!result.Contains(curve))
result.Add(curve);
}
}
return result;
}
// Powders that genuinely REQUIRE a clear coat say so explicitly. A casual "apply a clear coat
// for added durability" must NOT trip this — that over-flagged ~half the catalog and would pad
// quotes with unnecessary clear-coat steps.
private static readonly string[] RequiresClearPhrases =
{
"requires a clear", "requires clear", "require a clear",
"must be clear coated", "must be cleared", "needs a clear",
"clear coat is required", "clear coat required", "requires a clearcoat",
"requires a top coat", "clear coat to activate", "clear coat to achieve",
"requires a clear coat",
};
/// <summary>
/// Flags powders that genuinely need a clear coat: multi-step partial-cure (Illusion-style)
/// schedules, Columbia's named "Illusion" line, or explicit requirement phrasing. Casual
/// "you can clear coat this" mentions are intentionally ignored.
/// </summary>
public static bool DetectRequiresClearCoat(ColumbiaProduct p)
{
var cure = p.CureSchedule ?? string.Empty;
var name = p.Name ?? string.Empty;
// Partial-cure / multi-step instructions are the "apply this, then clear" case.
if (cure.Contains("partial cure", StringComparison.OrdinalIgnoreCase))
return true;
// Columbia's Illusion line needs a clear top coat to develop the effect.
if (name.Contains("Illusion", StringComparison.OrdinalIgnoreCase))
return true;
var text = $"{name} {cure} {p.Description}";
return RequiresClearPhrases.Any(phrase => text.Contains(phrase, StringComparison.OrdinalIgnoreCase));
}
// ── Misc field helpers ────────────────────────────────────────────────
/// <summary>Joins the color-group taxonomy ({name} entries) into a comma-separated families string.</summary>
public static string? BuildColorFamilies(ColumbiaProduct p)
{
var groups = p.PaColorGroup.Select(g => g.Name.Trim()).Where(n => n.Length > 0).Distinct().ToList();
if (groups.Count == 0)
{
// Fall back to the "Color Group" attribute options when the taxonomy is empty.
groups = p.Attributes
.Where(a => a.Name.Equals("Color Group", StringComparison.OrdinalIgnoreCase))
.SelectMany(a => a.Options.Select(o => o.Name.Trim()))
.Where(n => n.Length > 0)
.Distinct()
.ToList();
}
return groups.Count > 0 ? string.Join(",", groups) : null;
}
/// <summary>Normalizes resin chemistry — trims, and collapses the three Polyester/TGIC spellings.</summary>
public static string? NormalizeChemistry(string? type)
{
if (string.IsNullOrWhiteSpace(type))
return null;
var trimmed = type.Trim();
return PolyesterTgic.IsMatch(trimmed) ? "Polyester/TGIC" : trimmed;
}
/// <summary>Strips HTML tags/entities from a description and collapses whitespace to plain text.</summary>
public static string? StripHtml(string? html)
{
if (string.IsNullOrWhiteSpace(html))
return null;
var text = HtmlTag.Replace(html, " ");
text = WebUtility.HtmlDecode(text);
text = text.Replace("\r", " ").Replace("\n", " ").Replace("\t", " ");
text = WhitespaceRun.Replace(text, " ").Trim();
return text.Length > 0 ? text : null;
}
private static string? NullIfBlank(string? s) => string.IsNullOrWhiteSpace(s) ? null : s.Trim();
private static string? FirstNonBlank(params string?[] values) =>
values.FirstOrDefault(v => !string.IsNullOrWhiteSpace(v));
}
@@ -0,0 +1,187 @@
using System.Diagnostics;
using System.Globalization;
using Microsoft.Extensions.Logging;
using PowderCoating.Application.Constants;
using PowderCoating.Application.Interfaces;
using PowderCoating.Core.Entities;
using PowderCoating.Core.Interfaces;
namespace PowderCoating.Infrastructure.Services.Columbia;
/// <summary>
/// Full Columbia Coatings catalog sync: pages the API, maps each product, upserts via the shared
/// <see cref="IPowderCatalogUpsertService"/>, then reconciles discontinuations against the complete
/// pull. The discontinuation sweep runs ONLY after a successful full fetch — a partial pull (any
/// page failure throws from the client) aborts before the sweep so a transient error can never mass
/// flag the catalog as discontinued.
/// </summary>
public class ColumbiaCatalogSyncService : IColumbiaCatalogSyncService
{
private readonly IColumbiaCoatingsApiClient _client;
private readonly IPowderCatalogUpsertService _upsert;
private readonly IUnitOfWork _unitOfWork;
private readonly IPlatformSettingsService _settings;
private readonly ILogger<ColumbiaCatalogSyncService> _logger;
public ColumbiaCatalogSyncService(
IColumbiaCoatingsApiClient client,
IPowderCatalogUpsertService upsert,
IUnitOfWork unitOfWork,
IPlatformSettingsService settings,
ILogger<ColumbiaCatalogSyncService> logger)
{
_client = client;
_upsert = upsert;
_unitOfWork = unitOfWork;
_settings = settings;
_logger = logger;
}
/// <inheritdoc />
public async Task<ColumbiaSyncResult> RunSyncAsync(CancellationToken cancellationToken = default)
{
var result = new ColumbiaSyncResult { StartedAt = DateTime.UtcNow };
var stopwatch = Stopwatch.StartNew();
if (!_client.IsConfigured)
{
result.ErrorMessage = "Columbia API key is not configured.";
await RecordResultAsync(result);
return result;
}
try
{
// Full pull — throws on any page failure, which we treat as an incomplete sync.
var products = await _client.GetAllProductsAsync(cancellationToken);
result.TotalFetched = products.Count;
// Map and de-duplicate by (VendorName, SKU) in case the feed repeats a SKU.
// Exclude swatch cards and tester/sample size-variants — not standalone powder colors.
var mapped = products
.Where(p => !ColumbiaCatalogMapper.IsExcludedProduct(p))
.Select(ColumbiaCatalogMapper.Map)
.Where(m => !string.IsNullOrWhiteSpace(m.Sku))
.GroupBy(m => $"{m.VendorName}|{m.Sku}", StringComparer.OrdinalIgnoreCase)
.Select(g => g.First())
.ToList();
var upsertResult = await _upsert.UpsertAsync(mapped, result.StartedAt, cancellationToken);
result.Inserted = upsertResult.Inserted;
result.Updated = upsertResult.Updated;
result.Unchanged = upsertResult.Unchanged;
result.Skipped = upsertResult.Skipped;
// Remove any excluded records (swatches) that were synced before the exclusion existed,
// so they're deleted outright rather than lingering as "discontinued" powders.
await RemoveExcludedRecordsAsync();
// Complete pull succeeded — safe to reconcile discontinuations.
var incomingKeys = mapped
.Select(m => $"{m.VendorName}|{m.Sku}")
.ToHashSet(StringComparer.OrdinalIgnoreCase);
(result.Discontinued, result.Reactivated) =
await ReconcileDiscontinuationsAsync(incomingKeys, result.StartedAt);
result.Success = true;
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
_logger.LogError(ex, "Columbia catalog sync failed; skipping discontinuation sweep.");
result.Success = false;
result.ErrorMessage = ex.Message;
}
stopwatch.Stop();
result.Duration = stopwatch.Elapsed;
await RecordResultAsync(result);
return result;
}
/// <summary>
/// Flags catalog items sourced from Columbia that were NOT in this complete pull as discontinued,
/// and reactivates any previously-discontinued item that has reappeared. Returns (discontinued,
/// reactivated) counts.
/// </summary>
private async Task<(int Discontinued, int Reactivated)> ReconcileDiscontinuationsAsync(
HashSet<string> incomingKeys, DateTime runTimestamp)
{
var sourced = await _unitOfWork.PowderCatalog.FindAsync(
p => p.Source == ColumbiaIntegrationConstants.SourceName);
var discontinued = 0;
var reactivated = 0;
foreach (var item in sourced)
{
var present = incomingKeys.Contains($"{item.VendorName}|{item.Sku}");
if (!present && !item.IsDiscontinued)
{
item.IsDiscontinued = true;
item.UpdatedAt = runTimestamp;
await _unitOfWork.PowderCatalog.UpdateAsync(item);
discontinued++;
}
else if (present && item.IsDiscontinued)
{
item.IsDiscontinued = false;
item.UpdatedAt = runTimestamp;
await _unitOfWork.PowderCatalog.UpdateAsync(item);
reactivated++;
}
}
if (discontinued > 0 || reactivated > 0)
await _unitOfWork.CompleteAsync();
return (discontinued, reactivated);
}
/// <summary>
/// Deletes Columbia-sourced catalog rows that should not be in the catalog (swatch cards and
/// tester/sample size-variants). Mirrors <see cref="ColumbiaCatalogMapper.IsExcludedProduct"/>
/// on the stored columns. A no-op once the catalog is clean; guards against records synced
/// before the exclusion rule and ensures excluded items are removed, not flagged discontinued.
/// </summary>
private async Task RemoveExcludedRecordsAsync()
{
var excluded = (await _unitOfWork.PowderCatalog.FindAsync(p =>
p.Source == ColumbiaIntegrationConstants.SourceName
&& (p.Sku.EndsWith("-SW")
|| p.Sku.EndsWith("-04")
|| p.ColorName.Contains("SWATCH")
|| p.ColorName.Contains("Tester")
|| p.ColorName.Contains("Sample (")))).ToList();
if (excluded.Count == 0)
return;
foreach (var e in excluded)
await _unitOfWork.PowderCatalog.DeleteAsync(e);
await _unitOfWork.CompleteAsync();
_logger.LogInformation("Columbia sync: removed {Count} excluded record(s) (swatch/tester/sample) from the catalog.", excluded.Count);
}
/// <summary>Persists the run outcome to the last-synced / last-result platform settings.</summary>
private async Task RecordResultAsync(ColumbiaSyncResult result)
{
if (result.Success)
{
await _settings.SetAsync(
ColumbiaIntegrationConstants.SettingLastSyncedAt,
result.StartedAt.ToString("O", CultureInfo.InvariantCulture),
updatedBy: "Columbia Sync");
}
await _settings.SetAsync(
ColumbiaIntegrationConstants.SettingLastResult,
result.Summary,
updatedBy: "Columbia Sync");
}
}
@@ -0,0 +1,199 @@
using System.Net;
using System.Text.Json;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using PowderCoating.Application.Constants;
using PowderCoating.Application.DTOs.Columbia;
using PowderCoating.Application.Interfaces;
namespace PowderCoating.Infrastructure.Services;
/// <summary>
/// HTTP client for the Columbia Coatings product catalog API. Reads the API key and base URL from
/// configuration (<c>Columbia:ApiKey</c> / <c>Columbia:BaseUrl</c>), sends the <c>X-API-Key</c>
/// header, and pages the catalog via <c>GET /products</c>. Honors the documented rate limit
/// (120 requests / 60s) by retrying on HTTP 429 after the <c>Retry-After</c> interval.
/// </summary>
public class ColumbiaCoatingsApiClient : IColumbiaCoatingsApiClient
{
private const int MaxRetriesPer429 = 5;
private const int DefaultRetryAfterSeconds = 5;
private const int MaxPagesSafetyCap = 1000; // guards against a server that never reports last page
private readonly IHttpClientFactory _httpClientFactory;
private readonly IConfiguration _config;
private readonly ILogger<ColumbiaCoatingsApiClient> _logger;
/// <summary>
/// Columbia returns snake_case JSON; the snake-case naming policy binds it to our PascalCase DTOs
/// without per-property attributes. Case-insensitive as a belt-and-braces fallback.
/// </summary>
private static readonly JsonSerializerOptions JsonOptions = new()
{
PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower,
PropertyNameCaseInsensitive = true,
Converters = { new ColumbiaImageJsonConverter() },
};
public ColumbiaCoatingsApiClient(
IHttpClientFactory httpClientFactory,
IConfiguration config,
ILogger<ColumbiaCoatingsApiClient> logger)
{
_httpClientFactory = httpClientFactory;
_config = config;
_logger = logger;
}
private string? ApiKey => _config[ColumbiaIntegrationConstants.ConfigApiKey];
private string BaseUrl =>
(_config[ColumbiaIntegrationConstants.ConfigBaseUrl] ?? ColumbiaIntegrationConstants.DefaultBaseUrl)
.TrimEnd('/');
private string ApiBasePath =>
(_config[ColumbiaIntegrationConstants.ConfigApiBasePath] ?? ColumbiaIntegrationConstants.DefaultApiBasePath)
.Trim('/');
/// <summary>Fully-qualified products endpoint: host + configurable API base path + /products.</summary>
private string ProductsUrl => $"{BaseUrl}/{ApiBasePath}{ColumbiaIntegrationConstants.ProductsResource}";
public bool IsConfigured => !string.IsNullOrWhiteSpace(ApiKey);
/// <inheritdoc />
public async Task<ColumbiaProductsResponse> GetProductsPageAsync(
int page, int perPage, CancellationToken cancellationToken = default)
{
EnsureConfigured();
perPage = Math.Clamp(perPage, 1, ColumbiaIntegrationConstants.MaxPerPage);
var url = $"{ProductsUrl}?page={page}&per_page={perPage}";
var json = await SendWithRetryAsync(url, $"page {page}", cancellationToken);
if (json == null)
return new ColumbiaProductsResponse();
return JsonSerializer.Deserialize<ColumbiaProductsResponse>(json, JsonOptions) ?? new ColumbiaProductsResponse();
}
/// <inheritdoc />
public async Task<ColumbiaProduct?> GetProductBySkuAsync(string sku, CancellationToken cancellationToken = default)
{
EnsureConfigured();
if (string.IsNullOrWhiteSpace(sku))
return null;
var url = $"{ProductsUrl}?sku={Uri.EscapeDataString(sku)}&per_page=1";
var json = await SendWithRetryAsync(url, $"sku {sku}", cancellationToken);
if (json == null)
return null;
var response = JsonSerializer.Deserialize<ColumbiaProductsResponse>(json, JsonOptions);
return response?.Items.FirstOrDefault();
}
/// <inheritdoc />
public async Task<ColumbiaProduct?> GetProductByIdAsync(int id, CancellationToken cancellationToken = default)
{
EnsureConfigured();
// The by-id endpoint returns a bare product object (not the {items,pagination} envelope).
var url = $"{ProductsUrl}/{id}";
var json = await SendWithRetryAsync(url, $"id {id}", cancellationToken);
if (json == null)
return null;
return JsonSerializer.Deserialize<ColumbiaProduct>(json, JsonOptions);
}
/// <inheritdoc />
public async Task<IReadOnlyList<ColumbiaProduct>> GetAllProductsAsync(
CancellationToken cancellationToken = default)
{
EnsureConfigured();
var all = new List<ColumbiaProduct>();
for (var page = 1; page <= MaxPagesSafetyCap; page++)
{
var response = await GetProductsPageAsync(page, ColumbiaIntegrationConstants.MaxPerPage, cancellationToken);
if (response.Items.Count == 0)
break;
all.AddRange(response.Items);
// Stop when the pagination block says we've reached the last page.
if (response.Pagination is { TotalPages: > 0 } p && page >= p.TotalPages)
break;
}
_logger.LogInformation("Columbia API: retrieved {Count} products across paged requests.", all.Count);
return all;
}
/// <summary>Throws when no API key is configured so callers fail fast rather than 401.</summary>
private void EnsureConfigured()
{
if (!IsConfigured)
throw new InvalidOperationException("Columbia Coatings API key is not configured (Columbia:ApiKey).");
}
/// <summary>
/// Issues a GET with the API key header and returns the response body. Retries on HTTP 429
/// (honoring Retry-After) up to <see cref="MaxRetriesPer429"/>. Returns null on 404 so
/// single-product lookups surface "not found" without throwing; throws on any other non-success.
/// <paramref name="describe"/> is a short label (e.g. "page 3", "sku ABC") for log/error context.
/// </summary>
private async Task<string?> SendWithRetryAsync(string url, string describe, CancellationToken cancellationToken)
{
for (var attempt = 1; ; attempt++)
{
using var request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.Add("X-API-Key", ApiKey);
var client = _httpClientFactory.CreateClient();
using var response = await client.SendAsync(request, cancellationToken);
if (response.StatusCode == HttpStatusCode.TooManyRequests)
{
if (attempt > MaxRetriesPer429)
throw new HttpRequestException(
$"Columbia API still rate-limiting after {MaxRetriesPer429} retries ({describe}).");
var delaySeconds = GetRetryAfterSeconds(response) ?? DefaultRetryAfterSeconds;
_logger.LogWarning(
"Columbia API returned 429 ({Describe}, attempt {Attempt}); waiting {Delay}s before retry.",
describe, attempt, delaySeconds);
await Task.Delay(TimeSpan.FromSeconds(delaySeconds), cancellationToken);
continue;
}
if (response.StatusCode == HttpStatusCode.NotFound)
return null;
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync(cancellationToken);
}
}
/// <summary>
/// Parses the <c>Retry-After</c> header (delta-seconds or HTTP-date form) into whole seconds,
/// or null when absent/unparseable so the caller can fall back to a default.
/// </summary>
private static int? GetRetryAfterSeconds(HttpResponseMessage response)
{
var retryAfter = response.Headers.RetryAfter;
if (retryAfter == null)
return null;
if (retryAfter.Delta is { } delta)
return Math.Max(1, (int)Math.Ceiling(delta.TotalSeconds));
if (retryAfter.Date is { } date)
{
var seconds = (int)Math.Ceiling((date - DateTimeOffset.UtcNow).TotalSeconds);
return seconds > 0 ? seconds : 1;
}
return null;
}
}
@@ -6,6 +6,7 @@ using Anthropic.SDK.Messaging;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using PowderCoating.Application.Interfaces;
using PowderCoating.Core.Entities;
using PowderCoating.Core.Interfaces;
namespace PowderCoating.Infrastructure.Services;
@@ -541,18 +542,20 @@ Rules:
// Targeted prompt: we only need cure specs from this document
const string curePrompt = @"You are reading a Technical Data Sheet (TDS) for a powder coating product.
Extract ONLY the cure schedule. Respond with a valid JSON object — no markdown, no explanation:
Extract the cure schedule and the specific gravity. Respond with a valid JSON object — no markdown, no explanation:
{
""cureTemperatureF"": number or null,
""cureTimeMinutes"": number or null,
""reasoning"": ""one sentence: what cure schedule you found""
""specificGravity"": number or null,
""reasoning"": ""one sentence: what cure schedule and specific gravity you found""
}
Rules:
- cureTemperatureF: the recommended cure temperature in °F. Convert from °C if needed (multiply by 1.8 + 32). If a range is given use the midpoint. Most powders cure 325400 °F.
- cureTimeMinutes: the hold time at cure temperature in minutes (NOT total oven time). Typically 1020 min.
- If neither value can be found in the document, return null for both.";
- specificGravity: the specific gravity / density value from the TDS (often labeled ""Specific Gravity"" or ""Density""). Typically 1.21.8. Null if not stated.
- Return null for any value not found in the document.";
var sb = new StringBuilder();
sb.AppendLine("Technical Data Sheet content:");
@@ -603,11 +606,12 @@ Rules:
Success = true,
CureTemperatureF = GetDecimal(parsed, "cureTemperatureF"),
CureTimeMinutes = GetInt(parsed, "cureTimeMinutes"),
SpecificGravity = GetDecimal(parsed, "specificGravity"),
Reasoning = GetString(parsed, "reasoning"),
};
_logger.LogInformation("TDS cure lookup for {Url}: temp={Temp}°F, time={Time}min ({Reasoning})",
tdsUrl, result.CureTemperatureF, result.CureTimeMinutes, result.Reasoning);
_logger.LogInformation("TDS spec lookup for {Url}: temp={Temp}°F, time={Time}min, sg={Sg} ({Reasoning})",
tdsUrl, result.CureTemperatureF, result.CureTimeMinutes, result.SpecificGravity, result.Reasoning);
return result;
}
catch (Exception ex)
@@ -617,6 +621,58 @@ Rules:
}
}
/// <inheritdoc />
public async Task<bool> EnsureCatalogTdsSpecsAsync(PowderCatalogItem catalog)
{
// Already enriched, or nothing to read from. Specific gravity is the trigger: it's never in
// the API feed, so its absence means this item hasn't been TDS-enriched yet.
if (catalog == null || catalog.SpecificGravity.HasValue || string.IsNullOrWhiteSpace(catalog.TdsUrl))
return false;
var tds = await FetchTdsCureSpecsAsync(catalog.TdsUrl, catalog.ColorName);
if (!tds.Success)
return false;
var changed = false;
if (tds.SpecificGravity is > 0)
{
catalog.SpecificGravity = tds.SpecificGravity;
changed = true;
}
if (!catalog.CureTemperatureF.HasValue && tds.CureTemperatureF.HasValue)
{
catalog.CureTemperatureF = tds.CureTemperatureF;
changed = true;
}
if (!catalog.CureTimeMinutes.HasValue && tds.CureTimeMinutes.HasValue)
{
catalog.CureTimeMinutes = tds.CureTimeMinutes;
changed = true;
}
// Derive theoretical coverage once specific gravity is known.
if (!catalog.CoverageSqFtPerLb.HasValue && catalog.SpecificGravity is > 0)
{
catalog.CoverageSqFtPerLb = Math.Round(
TheoreticalCoverageConstant / (catalog.SpecificGravity.Value * DefaultCoverageThicknessMils),
2, MidpointRounding.AwayFromZero);
changed = true;
}
if (changed)
{
catalog.UpdatedAt = DateTime.UtcNow;
await _unitOfWork.PowderCatalog.UpdateAsync(catalog);
await _unitOfWork.CompleteAsync();
_logger.LogInformation(
"Lazily enriched catalog item {Vendor} {Sku} from TDS: sg={Sg}, cure={Temp}F/{Time}min, coverage={Cov}",
catalog.VendorName, catalog.Sku, catalog.SpecificGravity, catalog.CureTemperatureF,
catalog.CureTimeMinutes, catalog.CoverageSqFtPerLb);
}
return changed;
}
// ── Manufacturer URL pattern: build direct product page URL ───────────────
/// <summary>
@@ -0,0 +1,303 @@
using Microsoft.Extensions.Logging;
using PowderCoating.Application.Interfaces;
using PowderCoating.Core.Entities;
using PowderCoating.Core.Interfaces;
namespace PowderCoating.Infrastructure.Services;
/// <summary>
/// Single upsert path for the platform <see cref="PowderCatalogItem"/> master list, shared by the
/// JSON file import and the Columbia API sync. Match key is (VendorName, SKU), case-insensitive.
/// </summary>
public class PowderCatalogUpsertService : IPowderCatalogUpsertService
{
private readonly IUnitOfWork _unitOfWork;
private readonly ILogger<PowderCatalogUpsertService> _logger;
public PowderCatalogUpsertService(IUnitOfWork unitOfWork, ILogger<PowderCatalogUpsertService> logger)
{
_unitOfWork = unitOfWork;
_logger = logger;
}
/// <inheritdoc />
public async Task<PowderCatalogUpsertResult> UpsertAsync(
IReadOnlyList<PowderCatalogItem> incoming,
DateTime runTimestamp,
CancellationToken cancellationToken = default)
{
var result = new PowderCatalogUpsertResult();
// Load existing rows for just the vendors we're touching, keyed by (vendor|sku) lower-cased.
var vendorNames = incoming
.Select(i => i.VendorName)
.Where(v => !string.IsNullOrWhiteSpace(v))
.Distinct(StringComparer.OrdinalIgnoreCase)
.ToList();
var existing = (await _unitOfWork.PowderCatalog.FindAsync(p => vendorNames.Contains(p.VendorName)))
.ToDictionary(KeyOf, StringComparer.OrdinalIgnoreCase);
var toAdd = new List<PowderCatalogItem>();
foreach (var item in incoming)
{
if (string.IsNullOrWhiteSpace(item.Sku) || string.IsNullOrWhiteSpace(item.ColorName))
{
result.Skipped++;
continue;
}
if (existing.TryGetValue(KeyOf(item), out var record))
{
if (ApplyFeedFields(record, item))
{
record.UpdatedAt = runTimestamp;
record.LastSyncedAt = runTimestamp;
await _unitOfWork.PowderCatalog.UpdateAsync(record);
result.Updated++;
}
else
{
result.Unchanged++;
}
}
else
{
item.CreatedAt = runTimestamp;
item.LastSyncedAt = runTimestamp;
toAdd.Add(item);
result.Inserted++;
}
}
if (toAdd.Count > 0)
await _unitOfWork.PowderCatalog.AddRangeAsync(toAdd);
await _unitOfWork.CompleteAsync();
// Push current catalog price + product data down to any tenant inventory linked to these
// catalog rows, so quotes reflect the current price.
var propagated = await PropagateToLinkedInventoryAsync();
_logger.LogInformation(
"Powder catalog upsert: {Inserted} inserted, {Updated} updated, {Unchanged} unchanged, {Skipped} skipped; {Propagated} linked inventory item(s) refreshed.",
result.Inserted, result.Updated, result.Unchanged, result.Skipped, propagated);
return result;
}
/// <summary>
/// Keeps tenant inventory in step with the catalog (across all companies): first self-heals by
/// linking any unlinked item to its catalog row by identity, then refreshes every linked item
/// with the catalog's current price and product data. Returns the number of items touched.
/// </summary>
public async Task<int> PropagateToLinkedInventoryAsync()
{
var linkedCount = await LinkUnlinkedInventoryAsync();
var refreshedCount = await RefreshLinkedInventoryAsync();
return linkedCount + refreshedCount;
}
/// <summary>
/// Self-heals the catalog link: finds inventory items with no <see cref="InventoryItem.PowderCatalogItemId"/>
/// that match a catalog row by Manufacturer + ManufacturerPartNumber (the catalog SKU), sets the
/// FK, and applies the catalog price/product data. Only links on a confident match (exact SKU,
/// matching vendor, or a single unambiguous candidate) so it never mis-links. Returns the count
/// newly linked. This backfills items created before linking existed, on every environment, with
/// no manual step.
/// </summary>
private async Task<int> LinkUnlinkedInventoryAsync()
{
var unlinked = (await _unitOfWork.InventoryItems.FindAsync(
i => i.PowderCatalogItemId == null
&& i.Manufacturer != null && i.Manufacturer != ""
&& i.ManufacturerPartNumber != null && i.ManufacturerPartNumber != "",
ignoreQueryFilters: true)).ToList();
if (unlinked.Count == 0)
return 0;
var partNumbers = unlinked.Select(i => i.ManufacturerPartNumber!).Distinct().ToList();
var bySku = (await _unitOfWork.PowderCatalog.FindAsync(p => partNumbers.Contains(p.Sku)))
.GroupBy(c => c.Sku, StringComparer.OrdinalIgnoreCase)
.ToDictionary(g => g.Key, g => g.ToList(), StringComparer.OrdinalIgnoreCase);
if (bySku.Count == 0)
return 0;
var linked = 0;
foreach (var inv in unlinked)
{
if (!bySku.TryGetValue(inv.ManufacturerPartNumber!, out var candidates))
continue;
var mfr = inv.Manufacturer!.Trim().ToLower();
var match = candidates.FirstOrDefault(c => c.VendorName.ToLower().Contains(mfr))
?? (candidates.Count == 1 ? candidates[0] : null);
if (match == null)
continue;
inv.PowderCatalogItemId = match.Id;
ApplyCatalogToLinkedInventory(inv, match);
await _unitOfWork.InventoryItems.UpdateAsync(inv);
linked++;
}
if (linked > 0)
await _unitOfWork.CompleteAsync();
return linked;
}
/// <summary>
/// Refreshes every tenant inventory item linked to a powder catalog row (across all companies)
/// with the catalog's current list price and product data. Sets
/// <see cref="InventoryItem.CatalogReferencePrice"/> (the QUOTING price) and product spec/doc
/// fields, but NEVER the cost basis (UnitCost/AverageCost/LastPurchasePrice), quantity, notes,
/// image, location, or stock levels — those are tenant-owned. EF persists only items that
/// actually changed, so this is a cheap no-op when nothing moved. Returns the number updated.
/// </summary>
private async Task<int> RefreshLinkedInventoryAsync()
{
var linked = (await _unitOfWork.InventoryItems.FindAsync(
i => i.PowderCatalogItemId != null, ignoreQueryFilters: true)).ToList();
if (linked.Count == 0)
return 0;
var catalogIds = linked.Select(i => i.PowderCatalogItemId!.Value).Distinct().ToList();
var catalogById = (await _unitOfWork.PowderCatalog.FindAsync(p => catalogIds.Contains(p.Id)))
.ToDictionary(p => p.Id);
var updated = 0;
foreach (var inv in linked)
{
if (!catalogById.TryGetValue(inv.PowderCatalogItemId!.Value, out var cat))
continue;
if (ApplyCatalogToLinkedInventory(inv, cat))
{
await _unitOfWork.InventoryItems.UpdateAsync(inv);
updated++;
}
}
if (updated > 0)
await _unitOfWork.CompleteAsync();
return updated;
}
/// <summary>
/// Applies the catalog's current price and product data onto a linked inventory item, returning
/// true if anything changed. Sets the quoting reference price (only when the catalog has a real
/// price &gt; 0) and refreshes product/spec fields where the catalog has a value — never erasing
/// tenant data with catalog nulls, and never touching cost basis, quantity, notes, image, or
/// stock levels.
/// </summary>
private static bool ApplyCatalogToLinkedInventory(InventoryItem inv, PowderCatalogItem cat)
{
var changed = false;
// Quoting price (the point of this): keep the current catalog list price, separate from cost.
if (cat.UnitPrice > 0 && inv.CatalogReferencePrice != cat.UnitPrice)
{
inv.CatalogReferencePrice = cat.UnitPrice;
inv.CatalogPriceUpdatedAt = DateTime.UtcNow;
changed = true;
}
// Product data — refresh from the catalog where it has a value (catalog is authoritative on
// these); do not null out a tenant value the catalog doesn't carry.
changed |= SetStrIfCatalogHas(() => inv.Description, v => inv.Description = v, cat.Description);
changed |= SetStrIfCatalogHas(() => inv.Finish, v => inv.Finish = v, cat.Finish);
changed |= SetStrIfCatalogHas(() => inv.ColorFamilies, v => inv.ColorFamilies = v, cat.ColorFamilies);
changed |= SetStrIfCatalogHas(() => inv.SdsUrl, v => inv.SdsUrl = v, cat.SdsUrl);
changed |= SetStrIfCatalogHas(() => inv.TdsUrl, v => inv.TdsUrl = v, cat.TdsUrl);
changed |= SetStrIfCatalogHas(() => inv.SpecPageUrl, v => inv.SpecPageUrl = v, cat.ProductUrl);
if (cat.CureTemperatureF.HasValue && inv.CureTemperatureF != cat.CureTemperatureF)
{ inv.CureTemperatureF = cat.CureTemperatureF; changed = true; }
if (cat.CureTimeMinutes.HasValue && inv.CureTimeMinutes != cat.CureTimeMinutes)
{ inv.CureTimeMinutes = cat.CureTimeMinutes; changed = true; }
if (cat.CoverageSqFtPerLb.HasValue && inv.CoverageSqFtPerLb != cat.CoverageSqFtPerLb)
{ inv.CoverageSqFtPerLb = cat.CoverageSqFtPerLb; changed = true; }
if (cat.SpecificGravity.HasValue && inv.SpecificGravity != cat.SpecificGravity)
{ inv.SpecificGravity = cat.SpecificGravity; changed = true; }
if (cat.TransferEfficiency.HasValue && inv.TransferEfficiency != cat.TransferEfficiency)
{ inv.TransferEfficiency = cat.TransferEfficiency; changed = true; }
if (cat.RequiresClearCoat == true && !inv.RequiresClearCoat)
{ inv.RequiresClearCoat = true; changed = true; }
return changed;
}
/// <summary>Sets a string property from the catalog only when the catalog value is non-blank and differs.</summary>
private static bool SetStrIfCatalogHas(Func<string?> get, Action<string?> set, string? catalogValue)
{
if (!string.IsNullOrWhiteSpace(catalogValue) && !string.Equals(get(), catalogValue, StringComparison.Ordinal))
{
set(catalogValue);
return true;
}
return false;
}
private static string KeyOf(PowderCatalogItem p) => $"{p.VendorName}|{p.Sku}";
/// <summary>
/// Copies feed-sourced fields from <paramref name="src"/> onto <paramref name="dest"/> and
/// returns true if anything changed. Deliberately leaves enrichment fields (SpecificGravity,
/// CoverageSqFtPerLb, TransferEfficiency, Finish) and lifecycle flags untouched — those are
/// owned by lazy TDS/AI enrichment and the discontinuation sweep, not the feed.
/// </summary>
private static bool ApplyFeedFields(PowderCatalogItem dest, PowderCatalogItem src)
{
var changed = false;
changed |= Set(() => dest.ColorName, v => dest.ColorName = v, src.ColorName);
changed |= Set(() => dest.Description, v => dest.Description = v, src.Description);
changed |= src.UnitPrice > 0 && dest.UnitPrice != src.UnitPrice && Assign(() => dest.UnitPrice = src.UnitPrice);
changed |= Set(() => dest.PriceTiersJson, v => dest.PriceTiersJson = v, src.PriceTiersJson);
changed |= Set(() => dest.ImageUrl, v => dest.ImageUrl = v, src.ImageUrl);
changed |= Set(() => dest.SdsUrl, v => dest.SdsUrl = v, src.SdsUrl);
changed |= Set(() => dest.TdsUrl, v => dest.TdsUrl = v, src.TdsUrl);
changed |= Set(() => dest.ApplicationGuideUrl, v => dest.ApplicationGuideUrl = v, src.ApplicationGuideUrl);
changed |= Set(() => dest.ProductUrl, v => dest.ProductUrl = v, src.ProductUrl);
changed |= Set(() => dest.ChemistryType, v => dest.ChemistryType = v, src.ChemistryType);
changed |= Set(() => dest.MilThickness, v => dest.MilThickness = v, src.MilThickness);
changed |= Set(() => dest.CureScheduleText, v => dest.CureScheduleText = v, src.CureScheduleText);
changed |= Set(() => dest.CureCurvesJson, v => dest.CureCurvesJson = v, src.CureCurvesJson);
changed |= src.CureTemperatureF.HasValue && dest.CureTemperatureF != src.CureTemperatureF && Assign(() => dest.CureTemperatureF = src.CureTemperatureF);
changed |= src.CureTimeMinutes.HasValue && dest.CureTimeMinutes != src.CureTimeMinutes && Assign(() => dest.CureTimeMinutes = src.CureTimeMinutes);
changed |= src.RequiresClearCoat.HasValue && dest.RequiresClearCoat != src.RequiresClearCoat && Assign(() => dest.RequiresClearCoat = src.RequiresClearCoat);
changed |= Set(() => dest.ColorFamilies, v => dest.ColorFamilies = v, src.ColorFamilies);
changed |= Set(() => dest.FormulationChanges, v => dest.FormulationChanges = v, src.FormulationChanges);
changed |= Set(() => dest.Category, v => dest.Category = v, src.Category);
changed |= Set(() => dest.Source, v => dest.Source = v, src.Source);
return changed;
}
/// <summary>
/// Sets a nullable-string property when the feed provides a non-blank value that differs.
/// Merge semantics: a blank incoming value is ignored, so a partial feed (e.g. the Prismatic
/// file import, which omits cure/chemistry) never nulls out existing data.
/// </summary>
private static bool Set(Func<string?> get, Action<string?> set, string? newValue)
{
if (string.IsNullOrWhiteSpace(newValue))
return false;
if (!string.Equals(get(), newValue, StringComparison.Ordinal))
{
set(newValue);
return true;
}
return false;
}
/// <summary>Helper so a value assignment can participate in a boolean OR chain.</summary>
private static bool Assign(Action assign)
{
assign();
return true;
}
}
@@ -0,0 +1,112 @@
using System.Globalization;
using PowderCoating.Application.Constants;
using PowderCoating.Application.Interfaces;
namespace PowderCoating.Web.BackgroundServices;
/// <summary>
/// Runs the Columbia Coatings catalog sync on a schedule. Wakes hourly and triggers a full sync
/// only when the master switch (<c>ColumbiaSyncEnabled</c>) is on and the configured interval
/// (<c>ColumbiaSyncIntervalDays</c>) has elapsed since the last successful run. A full sync is
/// cheap (~25 API calls), so an hourly due-check is negligible; the actual work runs at most once
/// per interval. No-ops quietly when disabled or unconfigured.
/// </summary>
public class ColumbiaCatalogSyncBackgroundService : BackgroundService
{
private readonly IServiceScopeFactory _scopeFactory;
private readonly ILogger<ColumbiaCatalogSyncBackgroundService> _logger;
private static readonly TimeSpan CheckInterval = TimeSpan.FromHours(1);
private static readonly TimeSpan StartupDelay = TimeSpan.FromMinutes(2);
/// <summary>
/// Uses <see cref="IServiceScopeFactory"/> because a <see cref="BackgroundService"/> is a
/// singleton and the sync service / platform settings are scoped.
/// </summary>
public ColumbiaCatalogSyncBackgroundService(
IServiceScopeFactory scopeFactory,
ILogger<ColumbiaCatalogSyncBackgroundService> logger)
{
_scopeFactory = scopeFactory;
_logger = logger;
}
/// <inheritdoc />
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
_logger.LogInformation("ColumbiaCatalogSyncBackgroundService started.");
try
{
await Task.Delay(StartupDelay, stoppingToken);
while (!stoppingToken.IsCancellationRequested)
{
await RunIfDueAsync(stoppingToken);
await Task.Delay(CheckInterval, stoppingToken);
}
}
catch (OperationCanceledException)
{
// Shutting down — expected.
}
}
/// <summary>
/// Checks the enable switch and the elapsed interval, and runs a sync when due. Failures from
/// the sync itself are reported on its result (and recorded in platform settings) rather than
/// thrown, so a bad run never tears down the loop.
/// </summary>
private async Task RunIfDueAsync(CancellationToken ct)
{
using var scope = _scopeFactory.CreateScope();
var settings = scope.ServiceProvider.GetRequiredService<IPlatformSettingsService>();
if (!await settings.GetBoolAsync(ColumbiaIntegrationConstants.SettingEnabled))
return; // master switch off
var intervalDays = Math.Max(1, await settings.GetIntAsync(
ColumbiaIntegrationConstants.SettingIntervalDays,
ColumbiaIntegrationConstants.DefaultSyncIntervalDays));
if (!IsDue(await settings.GetAsync(ColumbiaIntegrationConstants.SettingLastSyncedAt), intervalDays))
return; // synced recently enough
var sync = scope.ServiceProvider.GetRequiredService<IColumbiaCatalogSyncService>();
try
{
_logger.LogInformation("Columbia scheduled sync starting (interval {Days}d).", intervalDays);
var result = await sync.RunSyncAsync(ct);
if (result.Success)
_logger.LogInformation("Columbia scheduled sync complete: {Summary}", result.Summary);
else
_logger.LogWarning("Columbia scheduled sync did not succeed: {Error}", result.ErrorMessage);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
_logger.LogError(ex, "Columbia scheduled sync threw unexpectedly.");
}
}
/// <summary>
/// A sync is due when there is no recorded last-sync timestamp, or the configured number of
/// days has elapsed since it. An unparseable timestamp is treated as "due".
/// </summary>
private static bool IsDue(string? lastSyncedRaw, int intervalDays)
{
if (string.IsNullOrWhiteSpace(lastSyncedRaw))
return true;
if (!DateTime.TryParse(lastSyncedRaw, CultureInfo.InvariantCulture,
DateTimeStyles.RoundtripKind, out var lastSynced))
return true;
return DateTime.UtcNow - lastSynced.ToUniversalTime() >= TimeSpan.FromDays(intervalDays);
}
}
@@ -25,6 +25,7 @@ public class DashboardController : Controller
private readonly ICompanyConfigHealthService _configHealth;
private readonly UserManager<ApplicationUser> _userManager;
private readonly ISubscriptionService _subscriptionService;
private readonly IInventoryAiLookupService _aiLookupService;
public DashboardController(
IUnitOfWork unitOfWork,
@@ -33,7 +34,8 @@ public class DashboardController : Controller
ITenantContext tenantContext,
ICompanyConfigHealthService configHealth,
UserManager<ApplicationUser> userManager,
ISubscriptionService subscriptionService)
ISubscriptionService subscriptionService,
IInventoryAiLookupService aiLookupService)
{
_unitOfWork = unitOfWork;
_logger = logger;
@@ -42,6 +44,7 @@ public class DashboardController : Controller
_configHealth = configHealth;
_userManager = userManager;
_subscriptionService = subscriptionService;
_aiLookupService = aiLookupService;
}
/// <summary>
@@ -765,59 +768,15 @@ public class DashboardController : Controller
UpdatedAt = DateTime.UtcNow,
};
await _unitOfWork.InventoryItems.AddAsync(inventoryItem);
await _unitOfWork.CompleteAsync(); // flush to get inventoryItem.Id
// Enrich from the platform powder catalog so the new inventory record carries the full
// spec/doc set (cure schedule, SDS/TDS, sample image, color families) rather than just
// the color code/name carried on the quote. Match by the catalog SKU (stored as the
// coat's colorCode), preferring the same manufacturer; fall back to color name.
await EnrichInventoryFromCatalogAsync(inventoryItem, colorCode, colorName, manufacturer);
// Opening stock transaction
var transaction = new InventoryTransaction
{
CompanyId = companyId,
InventoryItemId = inventoryItem.Id,
TransactionType = InventoryTransactionType.Purchase,
Quantity = lbsReceived,
UnitCost = unitCost ?? 0,
TotalCost = lbsReceived * (unitCost ?? 0),
TransactionDate = DateTime.UtcNow,
Notes = $"Initial stock — received from powder order for job {jobItem?.Job?.JobNumber}",
BalanceAfter = lbsReceived,
CreatedAt = DateTime.UtcNow,
UpdatedAt = DateTime.UtcNow,
};
await _unitOfWork.InventoryTransactions.AddAsync(transaction);
// Mark coat as received and link to the new inventory item
var userId = User.FindFirst(System.Security.Claims.ClaimTypes.NameIdentifier)?.Value;
coat.PowderReceived = true;
coat.PowderReceivedAt = DateTime.UtcNow;
coat.PowderReceivedByUserId = userId;
coat.PowderReceivedLbs = lbsReceived;
coat.InventoryItemId = inventoryItem.Id;
// Scan for sibling coats with the same custom powder and link them to the new item
var candidateCoats = await _unitOfWork.JobItemCoats.GetCandidateCoatsForLinkingAsync(coatId, companyId);
int linkedCount = 0;
foreach (var other in candidateCoats)
{
bool colorMatch = !string.IsNullOrWhiteSpace(colorCode)
? string.Equals(other.ColorCode?.Trim(), colorCode.Trim(), StringComparison.OrdinalIgnoreCase)
: !string.IsNullOrWhiteSpace(colorName) &&
string.Equals(other.ColorName?.Trim(), colorName.Trim(), StringComparison.OrdinalIgnoreCase);
if (!colorMatch) continue;
if (primaryVendorId.HasValue && other.VendorId.HasValue && other.VendorId != primaryVendorId)
continue;
other.InventoryItemId = inventoryItem.Id;
linkedCount++;
}
if (linkedCount > 0)
_logger.LogInformation("Linked {Count} additional coat(s) to new inventory item {ItemId} ({SKU})",
linkedCount, inventoryItem.Id, inventoryItem.SKU);
await _unitOfWork.CompleteAsync();
var linkedCount = await FinalizeReceivedPowderAsync(
coat, inventoryItem, lbsReceived, companyId, colorCode, colorName, primaryVendorId,
jobItem?.Job?.JobNumber);
return Json(new { success = true, itemName = inventoryItem.Name, sku = inventoryItem.SKU, linkedCount });
}
@@ -828,6 +787,267 @@ public class DashboardController : Controller
}
}
/// <summary>
/// Finds the platform powder catalog row for an inventory/coat identity: by catalog SKU
/// (stored as the coat's color code), preferring the same manufacturer, then by color name.
/// Returns null when no match is found.
/// </summary>
private async Task<PowderCatalogItem?> FindCatalogByIdentityAsync(
string? colorCode, string? colorName, string? manufacturer)
{
var code = colorCode?.Trim();
if (!string.IsNullOrWhiteSpace(code))
{
var codeLower = code.ToLower();
var hits = (await _unitOfWork.PowderCatalog.FindAsync(p => p.Sku.ToLower() == codeLower)).ToList();
var mfr = manufacturer?.Trim().ToLower();
var match = (!string.IsNullOrWhiteSpace(mfr)
? hits.FirstOrDefault(p => p.VendorName.ToLower().Contains(mfr))
: null)
?? hits.FirstOrDefault();
if (match != null)
return match;
}
if (!string.IsNullOrWhiteSpace(colorName))
{
var nameLower = colorName.Trim().ToLower();
return (await _unitOfWork.PowderCatalog.FindAsync(p => p.ColorName.ToLower() == nameLower))
.FirstOrDefault();
}
return null;
}
/// <summary>
/// Copies catalog spec/document fields onto an inventory item — cure schedule, coverage,
/// specific gravity, transfer efficiency, SDS/TDS links, sample image, color families, product
/// page — and links <see cref="InventoryItem.PowderCatalogItemId"/>. Only fills gaps, so any
/// value already set (e.g. entered on the receive form) is preserved.
/// </summary>
private static void ApplyCatalogToInventory(InventoryItem item, PowderCatalogItem catalog)
{
item.PowderCatalogItemId = catalog.Id;
if (string.IsNullOrWhiteSpace(item.ManufacturerPartNumber)) item.ManufacturerPartNumber = catalog.Sku;
if (string.IsNullOrWhiteSpace(item.Manufacturer)) item.Manufacturer = catalog.VendorName;
if (string.IsNullOrWhiteSpace(item.ColorName)) item.ColorName = catalog.ColorName;
if (string.IsNullOrWhiteSpace(item.Finish)) item.Finish = catalog.Finish;
if (string.IsNullOrWhiteSpace(item.ColorFamilies)) item.ColorFamilies = catalog.ColorFamilies;
if (string.IsNullOrWhiteSpace(item.ImageUrl)) item.ImageUrl = catalog.ImageUrl;
if (string.IsNullOrWhiteSpace(item.SdsUrl)) item.SdsUrl = catalog.SdsUrl;
if (string.IsNullOrWhiteSpace(item.TdsUrl)) item.TdsUrl = catalog.TdsUrl;
if (string.IsNullOrWhiteSpace(item.SpecPageUrl)) item.SpecPageUrl = catalog.ProductUrl;
item.CureTemperatureF ??= catalog.CureTemperatureF;
item.CureTimeMinutes ??= catalog.CureTimeMinutes;
item.SpecificGravity ??= catalog.SpecificGravity;
item.CoverageSqFtPerLb ??= catalog.CoverageSqFtPerLb ?? 30m;
item.TransferEfficiency ??= catalog.TransferEfficiency ?? 65m;
if (!item.RequiresClearCoat && catalog.RequiresClearCoat == true)
item.RequiresClearCoat = true;
if (item.UnitCost <= 0 && catalog.UnitPrice > 0)
{
item.UnitCost = catalog.UnitPrice;
item.LastPurchasePrice = catalog.UnitPrice;
}
// Quoting reference price (current catalog list price) — separate from cost basis above.
if (catalog.UnitPrice > 0)
{
item.CatalogReferencePrice = catalog.UnitPrice;
item.CatalogPriceUpdatedAt = DateTime.UtcNow;
}
}
/// <summary>
/// Fills blank spec/document fields on a received custom-powder inventory item from the matching
/// platform powder catalog row, so the tenant gets a complete record instead of just the color
/// code/name carried on the quote. No-op when the powder isn't in the catalog.
/// </summary>
private async Task EnrichInventoryFromCatalogAsync(
InventoryItem item, string? colorCode, string? colorName, string? manufacturer)
{
var catalog = await FindCatalogByIdentityAsync(colorCode, colorName, manufacturer);
if (catalog == null)
return;
// First use — lazily fill specific gravity / cure from the TDS before copying onto the item.
await _aiLookupService.EnsureCatalogTdsSpecsAsync(catalog);
ApplyCatalogToInventory(item, catalog);
}
/// <summary>
/// Shared finalize for a received powder: saves the inventory item, writes the opening Purchase
/// transaction, marks the coat received and links it, then links any sibling coats ordering the
/// same color. Returns the number of additional coats linked. Used by both the manual modal
/// (<see cref="AddCustomPowderToInventory"/>) and the catalog auto-receive
/// (<see cref="ReceivePowderFromCatalog"/>).
/// </summary>
private async Task<int> FinalizeReceivedPowderAsync(
JobItemCoat coat, InventoryItem inventoryItem, decimal lbsReceived, int companyId,
string? colorCode, string? colorName, int? primaryVendorId, string? jobNumber)
{
await _unitOfWork.InventoryItems.AddAsync(inventoryItem);
await _unitOfWork.CompleteAsync(); // flush to get inventoryItem.Id
var transaction = new InventoryTransaction
{
CompanyId = companyId,
InventoryItemId = inventoryItem.Id,
TransactionType = InventoryTransactionType.Purchase,
Quantity = lbsReceived,
UnitCost = inventoryItem.UnitCost,
TotalCost = lbsReceived * inventoryItem.UnitCost,
TransactionDate = DateTime.UtcNow,
Notes = $"Initial stock — received from powder order for job {jobNumber}",
BalanceAfter = lbsReceived,
CreatedAt = DateTime.UtcNow,
UpdatedAt = DateTime.UtcNow,
};
await _unitOfWork.InventoryTransactions.AddAsync(transaction);
var userId = User.FindFirst(System.Security.Claims.ClaimTypes.NameIdentifier)?.Value;
coat.PowderReceived = true;
coat.PowderReceivedAt = DateTime.UtcNow;
coat.PowderReceivedByUserId = userId;
coat.PowderReceivedLbs = lbsReceived;
coat.InventoryItemId = inventoryItem.Id;
var candidateCoats = await _unitOfWork.JobItemCoats.GetCandidateCoatsForLinkingAsync(coat.Id, companyId);
var linkedCount = 0;
foreach (var other in candidateCoats)
{
var colorMatch = !string.IsNullOrWhiteSpace(colorCode)
? string.Equals(other.ColorCode?.Trim(), colorCode.Trim(), StringComparison.OrdinalIgnoreCase)
: !string.IsNullOrWhiteSpace(colorName) &&
string.Equals(other.ColorName?.Trim(), colorName.Trim(), StringComparison.OrdinalIgnoreCase);
if (!colorMatch) continue;
if (primaryVendorId.HasValue && other.VendorId.HasValue && other.VendorId != primaryVendorId) continue;
other.InventoryItemId = inventoryItem.Id;
linkedCount++;
}
if (linkedCount > 0)
_logger.LogInformation("Linked {Count} additional coat(s) to new inventory item {ItemId} ({SKU})",
linkedCount, inventoryItem.Id, inventoryItem.SKU);
await _unitOfWork.CompleteAsync();
return linkedCount;
}
/// <summary>
/// Generates a unique powder SKU for a company in the form <c>{CODE}-{YYMM}-{####}</c>, where
/// CODE is the (padded) inventory category code. Mirrors the inventory SKU pattern used when
/// adding catalog-sourced powders.
/// </summary>
private async Task<string> GeneratePowderSkuAsync(InventoryCategoryLookup category)
{
var code = category.CategoryCode.Length >= 4
? category.CategoryCode[..4].ToUpperInvariant()
: category.CategoryCode.ToUpperInvariant().PadRight(4, 'X');
var yearMonth = DateTime.Now.ToString("yyMM");
var prefix = $"{code}-{yearMonth}-";
var allItems = await _unitOfWork.InventoryItems.GetAllAsync(ignoreQueryFilters: true);
var maxSeq = allItems
.Where(i => i.SKU.StartsWith(prefix))
.Select(i => int.TryParse(i.SKU[prefix.Length..], out var n) ? n : 0)
.DefaultIfEmpty(0)
.Max();
return $"{prefix}{(maxSeq + 1):D4}";
}
/// <summary>
/// Receives an ordered custom powder straight into inventory WITHOUT the manual modal when the
/// powder is already in the master catalog — the new record is fully populated from the catalog
/// (specs, SDS/TDS, image, pricing). Returns <c>needsDetails = true</c> (without saving) when
/// the powder isn't in the catalog or no coating category is configured, signaling the caller to
/// fall back to the manual entry modal.
/// </summary>
[HttpPost]
[ValidateAntiForgeryToken]
public async Task<IActionResult> ReceivePowderFromCatalog(int coatId, decimal lbsReceived)
{
try
{
if (lbsReceived <= 0)
return Json(new { success = false, message = "Quantity received must be greater than zero." });
var coat = await _unitOfWork.JobItemCoats.GetByIdAsync(coatId);
if (coat == null)
return Json(new { success = false, message = "Coat record not found." });
var jobItem = await _unitOfWork.JobItems.FirstOrDefaultAsync(
i => i.Coats.Any(c => c.Id == coatId), false, i => i.Job);
var companyId = jobItem?.Job?.CompanyId ?? _tenantContext.GetCurrentCompanyId() ?? 0;
// Only auto-receive when the powder resolves in the master catalog; otherwise the caller
// opens the manual modal.
var catalog = await FindCatalogByIdentityAsync(coat.ColorCode, coat.ColorName, null);
if (catalog == null)
return Json(new { success = false, needsDetails = true });
// First use — lazily fill specific gravity / cure from the TDS so the new record is complete.
await _aiLookupService.EnsureCatalogTdsSpecsAsync(catalog);
// Resolve the company's POWDER (coating) inventory category.
var categories = await _unitOfWork.InventoryCategoryLookups.FindAsync(c => c.CompanyId == companyId);
var coatingCategory = categories.FirstOrDefault(c => c.IsActive && c.IsCoating
&& c.CategoryCode.Equals("POWDER", StringComparison.OrdinalIgnoreCase))
?? categories.Where(c => c.IsActive && c.IsCoating)
.OrderBy(c => c.DisplayOrder).FirstOrDefault();
if (coatingCategory == null)
return Json(new { success = false, needsDetails = true });
var sku = await GeneratePowderSkuAsync(coatingCategory);
var inventoryItem = new InventoryItem
{
CompanyId = companyId,
SKU = sku,
Name = catalog.ColorName,
ColorName = catalog.ColorName,
ColorCode = coat.ColorCode,
InventoryCategoryId = coatingCategory.Id,
Category = coatingCategory.DisplayName,
QuantityOnHand = lbsReceived,
UnitOfMeasure = "lbs",
UnitCost = catalog.UnitPrice,
LastPurchasePrice = catalog.UnitPrice,
LastPurchaseDate = DateTime.UtcNow,
IsActive = true,
CreatedAt = DateTime.UtcNow,
UpdatedAt = DateTime.UtcNow,
};
ApplyCatalogToInventory(inventoryItem, catalog);
var linkedCount = await FinalizeReceivedPowderAsync(
coat, inventoryItem, lbsReceived, companyId, coat.ColorCode, coat.ColorName, null,
jobItem?.Job?.JobNumber);
return Json(new
{
success = true,
fromCatalog = true,
itemName = inventoryItem.Name,
sku = inventoryItem.SKU,
linkedCount
});
}
catch (Exception ex)
{
_logger.LogError(ex, "Error auto-receiving powder from catalog for coat {CoatId}", coatId);
return Json(new { success = false, message = "An error occurred while saving." });
}
}
/// <summary>
/// Platform-level dashboard visible only to SuperAdmins who are not impersonating a tenant.
/// Displays a cross-company overview: total/active/inactive company counts, user count,
@@ -240,6 +240,17 @@ public class InventoryController : Controller
var useMetric = await _tenantContext.UseMetricSystemAsync();
ViewBag.CoverageUnit = _measurementService.GetCoverageUnitLabel(useMetric);
// Manufacturer-level catalog status: prefer the linked catalog row, fall back to an
// identity match for items added before they were linked. Drives the "discontinued by
// manufacturer — cannot reorder" warning. This is distinct from the shop's own
// IsActive/DiscontinuedDate (whether the shop still stocks it).
var catalogItem = item.PowderCatalogItemId.HasValue
? await _unitOfWork.PowderCatalog.GetByIdAsync(item.PowderCatalogItemId.Value)
: await FindCatalogMatchAsync(item.Manufacturer, item.ManufacturerPartNumber);
ViewBag.CatalogDiscontinued = catalogItem?.IsDiscontinued ?? false;
ViewBag.CatalogVendorName = catalogItem?.VendorName;
ViewBag.CatalogProductUrl = catalogItem?.ProductUrl;
return View(itemDto);
}
catch (Exception ex)
@@ -302,6 +313,20 @@ public class InventoryController : Controller
item.Category = category.DisplayName;
}
// Link to the platform catalog row when this item's identity matches one, so the detail
// screen can show manufacturer-level status (discontinued / cannot reorder) and quotes
// can use the current catalog price.
var catalogMatch = await FindCatalogMatchAsync(item.Manufacturer, item.ManufacturerPartNumber);
if (catalogMatch != null)
{
item.PowderCatalogItemId = catalogMatch.Id;
if (catalogMatch.UnitPrice > 0)
{
item.CatalogReferencePrice = catalogMatch.UnitPrice;
item.CatalogPriceUpdatedAt = DateTime.UtcNow;
}
}
await _unitOfWork.InventoryItems.AddAsync(item);
await _unitOfWork.SaveChangesAsync();
@@ -763,6 +788,24 @@ public class InventoryController : Controller
/// Returns (wasInCatalog, addedToCatalog) so callers can surface UI badges.
/// Mutates <paramref name="result"/> in place.
/// </summary>
/// <summary>
/// Finds the platform powder catalog row matching an inventory item's identity
/// (Manufacturer + ManufacturerPartNumber), or null. Used to set
/// <see cref="InventoryItem.PowderCatalogItemId"/> and to surface manufacturer-level status
/// (e.g. discontinued / cannot reorder) on the detail screen.
/// </summary>
private async Task<PowderCatalogItem?> FindCatalogMatchAsync(string? manufacturer, string? sku)
{
if (string.IsNullOrWhiteSpace(manufacturer) || string.IsNullOrWhiteSpace(sku))
return null;
var skuLower = sku.Trim().ToLower();
var mfrLower = manufacturer.Trim().ToLower();
var hits = await _unitOfWork.PowderCatalog.FindAsync(p =>
p.Sku.ToLower() == skuLower && p.VendorName.ToLower().Contains(mfrLower));
return hits.FirstOrDefault();
}
private async Task<(bool wasInCatalog, bool addedToCatalog)> EnrichFromCatalogAsync(
InventoryAiLookupResult result, bool autoContribute)
{
@@ -1220,6 +1263,10 @@ public class InventoryController : Controller
if (catalogItem == null)
return Json(new { success = false, error = "Catalog item not found." });
// First use of this powder — lazily fill specific gravity / cure from its TDS so the new
// inventory record (and the catalog) carry complete specs. No-op once already enriched.
await _aiLookupService.EnsureCatalogTdsSpecsAsync(catalogItem);
var companyId = _tenantContext.GetCurrentCompanyId() ?? 0;
// Find the default coating category to assign.
@@ -1257,6 +1304,7 @@ public class InventoryController : Controller
ColorName = catalogItem.ColorName,
Manufacturer = catalogItem.VendorName,
ManufacturerPartNumber= catalogItem.Sku,
PowderCatalogItemId = catalogItem.Id,
Finish = catalogItem.Finish,
ColorFamilies = catalogItem.ColorFamilies,
RequiresClearCoat = catalogItem.RequiresClearCoat ?? false,
@@ -1272,6 +1320,8 @@ public class InventoryController : Controller
UnitCost = catalogItem.UnitPrice,
AverageCost = catalogItem.UnitPrice,
LastPurchasePrice = catalogItem.UnitPrice,
CatalogReferencePrice = catalogItem.UnitPrice > 0 ? catalogItem.UnitPrice : (decimal?)null,
CatalogPriceUpdatedAt = catalogItem.UnitPrice > 0 ? DateTime.UtcNow : (DateTime?)null,
QuantityOnHand = 0,
UnitOfMeasure = "lbs",
InventoryCategoryId = coatingCategory.Id,
@@ -1297,7 +1347,7 @@ public class InventoryController : Controller
efficiency = item.TransferEfficiency ?? 65m,
unitOfMeasure= item.UnitOfMeasure,
categoryName = coatingCategory.DisplayName,
costPerLb = item.UnitCost,
costPerLb = item.CatalogReferencePrice ?? item.UnitCost,
colorName = item.ColorName ?? item.Name,
colorCode = "",
isIncoming = true
@@ -3492,7 +3492,8 @@ public class JobsController : Controller
efficiency = i.TransferEfficiency ?? 65m,
unitOfMeasure = i.UnitOfMeasure ?? "lbs",
categoryName = i.InventoryCategory!.DisplayName,
costPerLb = i.UnitCost,
// Quote at the current catalog price when linked; fall back to their cost otherwise.
costPerLb = i.CatalogReferencePrice ?? i.UnitCost,
colorName = i.ColorName ?? i.Name,
colorCode = i.ColorCode ?? "",
isIncoming = i.IsIncoming
@@ -1,5 +1,6 @@
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
using PowderCoating.Application.Constants;
using PowderCoating.Application.DTOs.Common;
using PowderCoating.Application.DTOs.Inventory;
using PowderCoating.Application.Interfaces;
@@ -17,17 +18,31 @@ public class PowderCatalogController : Controller
{
private const decimal DefaultTransferEfficiency = 65m;
private const string JsonImportSource = "Manual JSON Import";
private readonly IUnitOfWork _unitOfWork;
private readonly IInventoryAiLookupService _aiLookupService;
private readonly IColumbiaCatalogSyncService _columbiaSyncService;
private readonly IPowderCatalogUpsertService _upsertService;
private readonly IPlatformSettingsService _platformSettings;
private readonly IConfiguration _config;
private readonly ILogger<PowderCatalogController> _logger;
public PowderCatalogController(
IUnitOfWork unitOfWork,
IInventoryAiLookupService aiLookupService,
IColumbiaCatalogSyncService columbiaSyncService,
IPowderCatalogUpsertService upsertService,
IPlatformSettingsService platformSettings,
IConfiguration config,
ILogger<PowderCatalogController> logger)
{
_unitOfWork = unitOfWork;
_aiLookupService = aiLookupService;
_columbiaSyncService = columbiaSyncService;
_upsertService = upsertService;
_platformSettings = platformSettings;
_config = config;
_logger = logger;
}
@@ -135,6 +150,11 @@ public class PowderCatalogController : Controller
}
};
// Columbia sync status for the admin panel (last run + master switch).
ViewBag.ColumbiaSyncEnabled = await _platformSettings.GetBoolAsync(ColumbiaIntegrationConstants.SettingEnabled);
ViewBag.ColumbiaLastSyncedAt = await _platformSettings.GetAsync(ColumbiaIntegrationConstants.SettingLastSyncedAt);
ViewBag.ColumbiaLastResult = await _platformSettings.GetAsync(ColumbiaIntegrationConstants.SettingLastResult);
return View(vm);
}
@@ -355,7 +375,8 @@ public class PowderCatalogController : Controller
PowderCatalogImportResult result;
try
{
result = await ImportJsonAsync(file, vendorName);
using var stream = file.OpenReadStream();
result = await ImportJsonAsync(stream, vendorName);
}
catch (Exception ex)
{
@@ -376,6 +397,67 @@ public class PowderCatalogController : Controller
return RedirectToAction(nameof(Index));
}
/// <summary>
/// Unattended catalog import for the offline scraper tool (e.g. PrismaticSync). Accepts the same
/// JSON scrape format in the request body, authenticated by a shared secret in the
/// <c>X-Import-Token</c> header (matched against <c>CatalogImport:Token</c>). The vendor name
/// comes from the <c>X-Vendor-Name</c> header. Runs through the same upsert as the manual upload.
/// Inert (401) until a token is configured.
/// </summary>
[HttpPost]
[AllowAnonymous]
[IgnoreAntiforgeryToken]
[RequestSizeLimit(50 * 1024 * 1024)] // 50 MB
public async Task<IActionResult> ImportApi()
{
var configuredToken = _config["CatalogImport:Token"];
if (string.IsNullOrWhiteSpace(configuredToken))
{
_logger.LogWarning("ImportApi called but no CatalogImport:Token is configured — rejecting.");
return Unauthorized(new { success = false, errorMessage = "Import API is not enabled." });
}
var providedToken = Request.Headers["X-Import-Token"].ToString();
if (!FixedTimeEquals(providedToken, configuredToken))
return Unauthorized(new { success = false, errorMessage = "Invalid import token." });
var vendorName = Request.Headers["X-Vendor-Name"].ToString();
if (string.IsNullOrWhiteSpace(vendorName))
vendorName = "Prismatic Powders";
try
{
var result = await ImportJsonAsync(Request.Body, vendorName);
_logger.LogInformation(
"ImportApi ({Vendor}): {Inserted} inserted, {Updated} updated, {Skipped} skipped, {Errors} errors.",
vendorName, result.Inserted, result.Updated, result.Skipped, result.Errors);
return Json(new
{
success = result.Success,
vendorName,
result.Inserted,
result.Updated,
result.Skipped,
result.Errors,
result.ErrorMessage
});
}
catch (Exception ex)
{
_logger.LogError(ex, "ImportApi failed for vendor {Vendor}", vendorName);
return StatusCode(500, new { success = false, errorMessage = "Import failed." });
}
}
/// <summary>Constant-time string comparison so token checks don't leak length/contents via timing.</summary>
private static bool FixedTimeEquals(string a, string b)
{
var ba = System.Text.Encoding.UTF8.GetBytes(a ?? string.Empty);
var bb = System.Text.Encoding.UTF8.GetBytes(b ?? string.Empty);
return System.Security.Cryptography.CryptographicOperations.FixedTimeEquals(ba, bb);
}
/// <summary>
/// AJAX endpoint used by the inventory form to search the catalog by SKU or color name.
/// SKU exact matches are ranked first; color name substring matches follow.
@@ -422,6 +504,78 @@ public class PowderCatalogController : Controller
return Json(results);
}
/// <summary>
/// Manually triggers a full Columbia Coatings catalog sync (SuperAdmin only). Bypasses the
/// scheduled interval. Reports the run outcome via TempData on the catalog index.
/// </summary>
[HttpPost]
[ValidateAntiForgeryToken]
public async Task<IActionResult> SyncColumbia(CancellationToken cancellationToken)
{
var result = await _columbiaSyncService.RunSyncAsync(cancellationToken);
if (result.Success)
TempData["Success"] = $"Columbia sync complete - {result.Summary}";
else
TempData["Error"] = $"Columbia sync failed: {result.ErrorMessage}";
return RedirectToAction(nameof(Index));
}
/// <summary>
/// Right-to-delete: removes every catalog record sourced from the Columbia Coatings API
/// (regardless of derived manufacturer, since PPG/KP products were served through that feed)
/// and nulls any inventory links to them across all tenants. The shops' own inventory stock
/// records survive — only the catalog link and discontinued badge are lost. SuperAdmin only.
/// </summary>
[HttpPost]
[ValidateAntiForgeryToken]
public async Task<IActionResult> PurgeColumbiaData()
{
var sourced = (await _unitOfWork.PowderCatalog.FindAsync(
p => p.Source == ColumbiaIntegrationConstants.SourceName)).ToList();
if (sourced.Count == 0)
{
TempData["Error"] = "There is no Columbia Coatings API data to remove.";
return RedirectToAction(nameof(Index));
}
var ids = sourced.Select(p => p.Id).ToList();
// Null the inventory links across ALL tenants (platform-level purge). A tenant's stock
// record is their data and must survive — it keeps its add-time snapshot, losing only the
// live catalog link.
var linked = (await _unitOfWork.InventoryItems.FindAsync(
i => i.PowderCatalogItemId.HasValue && ids.Contains(i.PowderCatalogItemId.Value),
ignoreQueryFilters: true)).ToList();
foreach (var inv in linked)
{
inv.PowderCatalogItemId = null;
await _unitOfWork.InventoryItems.UpdateAsync(inv);
}
foreach (var p in sourced)
await _unitOfWork.PowderCatalog.DeleteAsync(p);
await _unitOfWork.CompleteAsync();
// Reset sync tracking so the admin panel reflects the purge.
await _platformSettings.SetAsync(ColumbiaIntegrationConstants.SettingLastSyncedAt, null, "Columbia Purge");
await _platformSettings.SetAsync(
ColumbiaIntegrationConstants.SettingLastResult,
$"Purged {sourced.Count:N0} records on {DateTime.UtcNow:yyyy-MM-dd}",
"Columbia Purge");
_logger.LogWarning(
"Columbia data purge: deleted {Count} catalog records, unlinked {Linked} inventory items.",
sourced.Count, linked.Count);
TempData["Success"] =
$"Removed {sourced.Count:N0} Columbia Coatings catalog record(s) and unlinked " +
$"{linked.Count:N0} inventory item(s). Inventory stock was preserved.";
return RedirectToAction(nameof(Index));
}
// Private helpers
private async Task ApplyTdsCureFallbackAsync(InventoryAiLookupResult result, string? colorName)
@@ -438,9 +592,8 @@ public class PowderCatalogController : Controller
}
}
private async Task<PowderCatalogImportResult> ImportJsonAsync(IFormFile file, string vendorName)
private async Task<PowderCatalogImportResult> ImportJsonAsync(Stream stream, string vendorName)
{
using var stream = file.OpenReadStream();
using var doc = await JsonDocument.ParseAsync(stream);
if (!doc.RootElement.TryGetProperty("results", out var resultsEl) ||
@@ -449,13 +602,10 @@ public class PowderCatalogController : Controller
return new PowderCatalogImportResult { Success = false, ErrorMessage = "JSON must have a top-level 'results' array." };
}
// Load existing records for this vendor into a lookup dictionary
var existing = (await _unitOfWork.PowderCatalog.FindAsync(p => p.VendorName == vendorName))
.ToDictionary(p => p.Sku, StringComparer.OrdinalIgnoreCase);
var now = DateTime.UtcNow;
int inserted = 0, updated = 0, skipped = 0, errors = 0;
var toAdd = new List<PowderCatalogItem>();
// Map the scrape format to catalog items, then hand off to the shared upsert path (same
// one the Columbia API sync uses) so there is a single insert/update/diff implementation.
var mapped = new List<PowderCatalogItem>();
int skipped = 0, errors = 0;
foreach (var item in resultsEl.EnumerateArray())
{
@@ -469,49 +619,21 @@ public class PowderCatalogController : Controller
continue;
}
var rawDesc = item.GetStringOrNull("description");
var cleanDesc = StripBoilerplate(rawDesc);
var unitPrice = ExtractBasePrice(item);
var priceTiersJson = item.TryGetProperty("price_tiers", out var tiersEl)
? tiersEl.GetRawText()
: null;
if (existing.TryGetValue(sku, out var record))
mapped.Add(new PowderCatalogItem
{
record.ColorName = colorName;
record.Description = cleanDesc;
record.UnitPrice = unitPrice;
record.PriceTiersJson = priceTiersJson;
record.ImageUrl = item.GetStringOrNull("sample_image_url");
record.SdsUrl = item.GetStringOrNull("safety_data_sheet_url");
record.TdsUrl = item.GetStringOrNull("technical_data_sheet_url");
record.ApplicationGuideUrl = item.GetStringOrNull("application_guide_url");
record.ProductUrl = item.GetStringOrNull("product_url");
record.UpdatedAt = now;
record.LastSyncedAt = now;
await _unitOfWork.PowderCatalog.UpdateAsync(record);
updated++;
}
else
{
toAdd.Add(new PowderCatalogItem
{
VendorName = vendorName,
Sku = sku,
ColorName = colorName,
Description = cleanDesc,
UnitPrice = unitPrice,
PriceTiersJson = priceTiersJson,
ImageUrl = item.GetStringOrNull("sample_image_url"),
SdsUrl = item.GetStringOrNull("safety_data_sheet_url"),
TdsUrl = item.GetStringOrNull("technical_data_sheet_url"),
ApplicationGuideUrl = item.GetStringOrNull("application_guide_url"),
ProductUrl = item.GetStringOrNull("product_url"),
CreatedAt = now,
LastSyncedAt = now
});
inserted++;
}
VendorName = vendorName,
Source = JsonImportSource,
Sku = sku,
ColorName = colorName,
Description = StripBoilerplate(item.GetStringOrNull("description")),
UnitPrice = ExtractBasePrice(item),
PriceTiersJson = item.TryGetProperty("price_tiers", out var tiersEl) ? tiersEl.GetRawText() : null,
ImageUrl = item.GetStringOrNull("sample_image_url"),
SdsUrl = item.GetStringOrNull("safety_data_sheet_url"),
TdsUrl = item.GetStringOrNull("technical_data_sheet_url"),
ApplicationGuideUrl = item.GetStringOrNull("application_guide_url"),
ProductUrl = item.GetStringOrNull("product_url"),
});
}
catch (Exception ex)
{
@@ -520,17 +642,14 @@ public class PowderCatalogController : Controller
}
}
if (toAdd.Any())
await _unitOfWork.PowderCatalog.AddRangeAsync(toAdd);
await _unitOfWork.CompleteAsync();
var upsert = await _upsertService.UpsertAsync(mapped, DateTime.UtcNow);
return new PowderCatalogImportResult
{
Success = true,
Inserted = inserted,
Updated = updated,
Skipped = skipped,
Inserted = upsert.Inserted,
Updated = upsert.Updated,
Skipped = skipped + upsert.Skipped,
Errors = errors
};
}
@@ -2545,7 +2545,8 @@ public class QuotesController : Controller
efficiency = i.TransferEfficiency ?? 65m,
unitOfMeasure = i.UnitOfMeasure ?? "lbs",
categoryName = i.InventoryCategory!.DisplayName,
costPerLb = i.UnitCost,
// Quote at the current catalog price when linked; fall back to their cost otherwise.
costPerLb = i.CatalogReferencePrice ?? i.UnitCost,
colorName = i.ColorName ?? i.Name,
colorCode = i.ColorCode ?? "",
isIncoming = i.IsIncoming
@@ -481,6 +481,8 @@ public static class HelpKnowledgeBase
5. Enter opening quantity on hand the system automatically records an Initial transaction for audit purposes
6. Save
**Manufacturer catalog integration:** The platform is integrated with the Columbia Coatings product catalog. When you add a Columbia powder it auto-fills the color, specs, cure schedule, and SDS/TDS links, and keeps the price current catalog data refreshes regularly (near real-time). Quotes use the latest catalog price even when your stored cost is older, and a powder's detail page shows the current catalog price (and flags when it has changed since you last bought it). Discontinued powders are flagged "cannot reorder" but stay usable for stock you already have.
**Stock status:** Three states are shown on every item:
- **In Stock** (green) quantity is above the reorder point
- **Low Stock** (red) quantity is greater than zero but at or below the reorder point; time to reorder
+5
View File
@@ -11,6 +11,7 @@ using PowderCoating.Core.Interfaces.Services;
using PowderCoating.Infrastructure.Data;
using PowderCoating.Infrastructure.Repositories;
using PowderCoating.Infrastructure.Services;
using PowderCoating.Infrastructure.Services.Columbia;
using PowderCoating.Application.Interfaces;
using PowderCoating.Application.Services;
using PowderCoating.Application.Configuration;
@@ -222,6 +223,9 @@ builder.Services.AddScoped<IInventoryAiLookupService, InventoryAiLookupService>(
builder.Services.AddScoped<ICustomFormulaAiService, CustomFormulaAiService>();
builder.Services.AddScoped<IFormulaLibraryService, FormulaLibraryService>();
builder.Services.AddHttpClient();
builder.Services.AddScoped<IColumbiaCoatingsApiClient, ColumbiaCoatingsApiClient>();
builder.Services.AddScoped<IPowderCatalogUpsertService, PowderCatalogUpsertService>();
builder.Services.AddScoped<IColumbiaCatalogSyncService, ColumbiaCatalogSyncService>();
builder.Services.AddScoped<ICompanyLogoService, CompanyLogoService>();
builder.Services.AddScoped<IEquipmentManualService, EquipmentManualService>();
builder.Services.AddScoped<IPricingCalculationService, PricingCalculationService>();
@@ -255,6 +259,7 @@ builder.Services.AddHostedService<StripeWebhookRetentionBackgroundService>();
builder.Services.AddHostedService<SetupWizardReminderBackgroundService>();
builder.Services.AddHostedService<RecurringTransactionService>();
builder.Services.AddHostedService<AppointmentReminderBackgroundService>();
builder.Services.AddHostedService<ColumbiaCatalogSyncBackgroundService>();
builder.Services.AddScoped<ISubscriptionService, SubscriptionService>();
builder.Services.AddScoped<IStripeService, StripeService>();
builder.Services.AddScoped<IStripeConnectService, StripeConnectService>();
@@ -1041,6 +1041,37 @@
// Custom powder (no inventory item) â†' open modal to add to inventory
if (!hasInv) {
// If the powder is already in the master catalog, receive it straight to inventory
// with all its specs/docs — no modal. Only fall back to the modal when it isn't.
const tokenAuto = document.querySelector('input[name="__RequestVerificationToken"]')?.value
?? document.querySelector('meta[name="__RequestVerificationToken"]')?.content;
this.disabled = true; qtyInput.disabled = true;
this.innerHTML = '<span class="spinner-border spinner-border-sm"></span>';
try {
const autoResp = await fetch('@Url.Action("ReceivePowderFromCatalog", "Dashboard")', {
method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded', 'RequestVerificationToken': tokenAuto },
body: `coatId=${coatId}&lbsReceived=${lbs}`
});
const autoData = await autoResp.json();
if (autoData.success) {
fadePlacedRow(row);
showInventoryToast('Added "' + (autoData.itemName || 'powder') + '" to inventory from the catalog.');
return;
}
if (!autoData.needsDetails) {
alert(autoData.message || 'Could not record receipt. Please try again.');
this.disabled = false; qtyInput.disabled = false;
this.innerHTML = '<i class="bi bi-box-arrow-in-down"></i> Got It';
return;
}
// Not in catalog — fall through to the manual entry modal.
} catch {
// Network error — fall back to the manual entry modal.
}
this.disabled = false; qtyInput.disabled = false;
this.innerHTML = '<i class="bi bi-box-arrow-in-down"></i> Got It';
const modal = document.getElementById('addPowderModal');
// Pre-fill hidden + text fields
modal.querySelector('#apm-coatId').value = coatId;
@@ -138,6 +138,18 @@
<li class="mb-1">If a vendor name is selected in the Vendor field before searching, results are scoped to that vendor first, then broadened automatically if nothing matches.</li>
</ul>
<div class="alert alert-permanent alert-success d-flex gap-2 mb-3" role="alert">
<i class="bi bi-cloud-check me-1 flex-shrink-0 mt-1"></i>
<div>
<strong>Columbia Coatings integration:</strong> the catalog is connected directly to the Columbia Coatings
product catalog and refreshes regularly (near real-time). Columbia powders auto-fill their full specs, cure
schedule, and SDS/TDS links, and their prices stay current &mdash; quotes use the latest catalog price even
when your stored cost is older. An item&rsquo;s detail page shows the current catalog price and flags when it
has changed since you last bought it. Discontinued powders are flagged &ldquo;cannot reorder&rdquo; but stay
usable for stock you already have.
</div>
</div>
<h3 class="h6 fw-semibold mt-4 mb-2"><i class="bi bi-camera me-1"></i>Label Scanner (Camera)</h3>
<p>
Click the <strong>camera icon</strong> next to the Lookup button to open the label scanner.
@@ -69,6 +69,12 @@
{
<span class="badge bg-danger"><i class="bi bi-x-circle me-1"></i>Inactive</span>
}
@if ((bool?)ViewBag.CatalogDiscontinued == true)
{
<span class="badge bg-warning text-dark" title="Discontinued by the manufacturer — cannot reorder">
<i class="bi bi-slash-circle me-1"></i>Discontinued by manufacturer
</span>
}
</div>
</div>
<div class="d-flex gap-2">
@@ -103,6 +109,20 @@
<div><strong>Status:</strong> This item is inactive</div>
</div>
}
@if ((bool?)ViewBag.CatalogDiscontinued == true)
{
<div class="alert alert-warning alert-permanent d-flex align-items-center mb-3">
<i class="bi bi-slash-circle me-2"></i>
<div>
<strong>Discontinued by @(ViewBag.CatalogVendorName ?? "manufacturer"):</strong>
this powder has been discontinued and cannot be reordered. Existing stock can still be used and quoted.
@if (!string.IsNullOrEmpty(ViewBag.CatalogProductUrl as string))
{
<a href="@ViewBag.CatalogProductUrl" target="_blank" rel="noopener" class="alert-link ms-1">View product page</a>
}
</div>
</div>
}
<div class="row g-4">
<!-- Left Column -->
@@ -411,6 +431,41 @@
<label class="text-muted small mb-1">Total Stock Value</label>
<p class="fw-semibold text-primary mb-0 fs-5">@((Model.QuantityOnHand * Model.UnitCost).ToString("C"))</p>
</div>
@if (Model.CatalogReferencePrice.HasValue && Model.CatalogReferencePrice.Value > 0)
{
<div class="col-12"><hr class="my-2" /></div>
<div class="col-12">
<label class="text-muted small mb-1">
Current Catalog Price
<i class="bi bi-info-circle ms-1" role="button" tabindex="0"
data-bs-toggle="popover" data-bs-trigger="hover focus" data-bs-placement="top"
data-bs-content="The current list price from the linked manufacturer catalog, refreshed by sync. New quotes use this price. It does not change your Unit Cost or stock value."></i>
</label>
<p class="fw-semibold text-success mb-0 fs-5">
@Model.CatalogReferencePrice.Value.ToString("C")
<span class="text-muted fs-6 fw-normal">/ @Model.UnitOfMeasure</span>
</p>
@{
var catRef = Model.CatalogReferencePrice.Value;
var paidPrice = (Model.LastPurchaseDate.HasValue && Model.LastPurchasePrice > 0)
? Model.LastPurchasePrice : Model.UnitCost;
}
@if (paidPrice > 0 && Math.Abs(catRef - paidPrice) >= 0.01m)
{
var priceUp = catRef > paidPrice;
<div class="mt-1">
<span class="badge @(priceUp ? "bg-warning text-dark" : "bg-info text-dark")">
<i class="bi @(priceUp ? "bi-arrow-up-right" : "bi-arrow-down-right") me-1"></i>
Price @(priceUp ? "up" : "down") from @paidPrice.ToString("C") last paid
</span>
</div>
}
@if (Model.CatalogPriceUpdatedAt.HasValue)
{
<div class="text-muted small mt-1">Updated @Model.CatalogPriceUpdatedAt.Value.ToLocalTime().ToString("MMM d, yyyy")</div>
}
</div>
}
@if (Model.LastPurchaseDate.HasValue)
{
<div class="col-6">
@@ -110,12 +110,49 @@
<div class="text-muted small">Platform-level lookup library for inventory autofill, SDS/TDS links, and curing specs.</div>
</div>
<div class="d-flex gap-2">
<form asp-action="SyncColumbia" method="post" class="d-inline"
onsubmit="this.querySelector('button').disabled=true;this.querySelector('button').innerHTML='<span class=\'spinner-border spinner-border-sm me-1\'></span>Syncing&hellip;';">
@Html.AntiForgeryToken()
<button type="submit" class="btn btn-outline-primary" title="Pull the latest Columbia Coatings catalog now">
<i class="bi bi-cloud-download me-1"></i>Sync Columbia
</button>
</form>
<a asp-action="Create" class="btn btn-primary">
<i class="bi bi-plus-circle me-1"></i>Add Powder
</a>
</div>
</div>
@{
var columbiaLastSynced = ViewBag.ColumbiaLastSyncedAt as string;
var columbiaLastResult = ViewBag.ColumbiaLastResult as string;
var columbiaEnabled = ViewBag.ColumbiaSyncEnabled is true;
}
<div class="alert alert-light border d-flex flex-wrap align-items-center gap-2 small mb-4 alert-permanent">
<span class="badge @(columbiaEnabled ? "bg-success" : "bg-secondary")">
Scheduled sync @(columbiaEnabled ? "on" : "off")
</span>
@if (!string.IsNullOrWhiteSpace(columbiaLastSynced) && DateTime.TryParse(columbiaLastSynced, out var lastSyncedAt))
{
<span class="text-muted">Last synced @lastSyncedAt.ToLocalTime().ToString("MMM d, yyyy h:mm tt")</span>
}
else
{
<span class="text-muted">Never synced</span>
}
@if (!string.IsNullOrWhiteSpace(columbiaLastResult))
{
<span class="text-muted">&mdash; @columbiaLastResult</span>
}
<form asp-action="PurgeColumbiaData" method="post" class="ms-auto"
onsubmit="return confirm('Remove ALL Columbia Coatings API data from the powder catalog? This deletes every record sourced from their feed (including PPG and KP Pigments products served through it) and unlinks inventory items. Inventory stock is preserved. This cannot be undone.');">
@Html.AntiForgeryToken()
<button type="submit" class="btn btn-outline-danger btn-sm" title="Right-to-delete: remove all Columbia-sourced catalog data">
<i class="bi bi-trash me-1"></i>Remove Columbia data
</button>
</form>
</div>
<div class="row g-3 mb-4 powder-catalog-summary">
<div class="col-sm-6 col-xl-2">
<div class="card h-100">
+8
View File
@@ -42,6 +42,14 @@
"ApiKey": "25651af3a4829559ef0dfa1758fa3edbf92b6b76"
}
},
"Columbia": {
"ApiKey": "cca_live_ffd5e355809e1d23007d82982684157de5727c226bc2b482",
"BaseUrl": "https://columbiacoatings.com",
"ApiBasePath": "/wp-json/cca/v1"
},
"CatalogImport": {
"Token": ""
},
"SendGrid": {
"ApiKey": "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",
"FromEmail": "spouliot@scppowdercoating.com",
@@ -0,0 +1,323 @@
using System.Text.Json;
using PowderCoating.Application.DTOs.Columbia;
using PowderCoating.Infrastructure.Services.Columbia;
namespace PowderCoating.UnitTests;
/// <summary>
/// Tests for the Columbia catalog mapper, focused on the fields that are tricky in the real feed:
/// the free-text cure parser (multiple glyphs, multi-curve, partial-cure), manufacturer derivation
/// from a multi-brand distributor, pricing across simple/variable products, and HTML stripping.
/// Cases mirror records captured from the live API.
/// </summary>
public class ColumbiaCatalogMapperTests
{
// ── Cure schedule parsing ─────────────────────────────────────────────
[Fact]
public void ParseCureCurves_SimpleSchedule_ReturnsSingleCurve()
{
var curves = ColumbiaCatalogMapper.ParseCureCurves("10 minutes @ 400°F");
Assert.Single(curves);
Assert.Equal(400, curves[0].TempF);
Assert.Equal(10, curves[0].Minutes);
}
[Fact]
public void ParseCureCurves_MetalTemperaturePrefixWithCelsius_ParsesFahrenheitOnly()
{
var curves = ColumbiaCatalogMapper.ParseCureCurves("Metal Temperature: 10 minutes at 400°F (204°C)");
Assert.Single(curves);
Assert.Equal(400, curves[0].TempF);
Assert.Equal(10, curves[0].Minutes);
}
[Theory]
[InlineData("10 minutes @ 400˚F (204˚C)")] // U+02DA ring above
[InlineData("10 minutes @ 400ºF (204ºC)")] // U+00BA masculine ordinal
[InlineData("Metal temperature: 10 minutes @ 400F (204C)")] // no degree glyph at all
public void ParseCureCurves_DegreeGlyphVariants_AllParse(string schedule)
{
var curves = ColumbiaCatalogMapper.ParseCureCurves(schedule);
Assert.Single(curves);
Assert.Equal(400, curves[0].TempF);
}
[Fact]
public void ParseCureCurves_MultiCurve_CapturesAllInOrder_PrimaryIsFirst()
{
// Real record S1790085-55: standard high-temp curve first, low-temp alternates after.
var schedule = "Metal Temperature: 5 minutes at 400°F (204°C) -or- 10 minutes at 360°F (182°C)* 15 minutes at 340°F (171°C)* *Low-Temp cure curve";
var curves = ColumbiaCatalogMapper.ParseCureCurves(schedule);
Assert.Equal(3, curves.Count);
Assert.Equal(new ColumbiaCatalogMapper.CureCurve(400, 5), curves[0]); // primary
Assert.Equal(new ColumbiaCatalogMapper.CureCurve(360, 10), curves[1]); // alternate
Assert.Equal(new ColumbiaCatalogMapper.CureCurve(340, 15), curves[2]); // alternate
}
[Fact]
public void ParseCureCurves_PartialCureInstructions_ReturnsEmpty()
{
// Illusion powder F1697027 — multi-step, no single temp/time pair.
var schedule = "(1) - Apply a basecoat and partial cure. (2) - Apply this powder and partial cure (3). - Apply a clear coat and fully cure.";
var curves = ColumbiaCatalogMapper.ParseCureCurves(schedule);
Assert.Empty(curves);
}
[Fact]
public void Map_MultiCurve_SetsPrimaryTempTimeAndStoresAllCurvesJson()
{
var product = new ColumbiaProduct
{
Sku = "S1790085-55",
Name = "Multi Cure",
CureSchedule = "5 minutes at 400°F -or- 10 minutes at 360°F",
};
var item = ColumbiaCatalogMapper.Map(product);
Assert.Equal(400m, item.CureTemperatureF);
Assert.Equal(5, item.CureTimeMinutes);
Assert.False(string.IsNullOrEmpty(item.CureCurvesJson));
Assert.Equal("5 minutes at 400°F -or- 10 minutes at 360°F", item.CureScheduleText);
var curves = JsonSerializer.Deserialize<List<ColumbiaCatalogMapper.CureCurve>>(item.CureCurvesJson!);
Assert.Equal(2, curves!.Count);
}
// ── Manufacturer derivation ───────────────────────────────────────────
[Fact]
public void DeriveManufacturer_AddSkuPrefix_IsKpPigments()
{
var p = new ColumbiaProduct { Sku = "ADD-BRBDS", Name = "Barbados Blue ColorShift Pearl" };
Assert.Equal("KP Pigments", ColumbiaCatalogMapper.DeriveManufacturer(p));
Assert.True(ColumbiaCatalogMapper.IsAdditive(p));
}
[Fact]
public void DeriveManufacturer_PpgCategory_IsPpg()
{
var p = new ColumbiaProduct
{
Sku = "PCU75139",
Name = "PPG Chrome Shadow",
Categories = { new ColumbiaNamed { Name = "PPG Powders" } },
};
Assert.Equal("PPG", ColumbiaCatalogMapper.DeriveManufacturer(p));
}
[Fact]
public void DeriveManufacturer_HouseBrand_IsColumbia()
{
var p = new ColumbiaProduct
{
Sku = "X5004124",
Name = "Blue Wrinkle",
Categories = { new ColumbiaNamed { Name = "Powders" }, new ColumbiaNamed { Name = "New Releases" } },
};
Assert.Equal("Columbia Coatings", ColumbiaCatalogMapper.DeriveManufacturer(p));
Assert.False(ColumbiaCatalogMapper.IsAdditive(p));
}
// ── Excluded products (swatches + tester/sample size variants) ────────
[Theory]
[InlineData("T1696049-SW", "**SWATCH** - Copperhead II", true)] // swatch card
[InlineData("T1696049-SW", "Copperhead II", true)] // -SW suffix alone
[InlineData("X1", "**SWATCH** - Something", true)] // name marker alone
[InlineData("T1696049-04", "Copperhead II (4 Ounce Tester)", true)] // tester = size variant
[InlineData("XYZ", "Copperhead II (4 Ounce Tester)", true)] // tester by name
[InlineData("S1760090-S", "Black Beauty Sample (5 lbs)", true)] // 5lb sample = size variant
[InlineData("X5004124", "Blue Wrinkle", false)] // normal powder
[InlineData("S5704126", "Smokey Blue", false)] // normal powder, SKU starts with S
public void IsExcludedProduct_DetectsSwatchesTestersAndSamples(string sku, string name, bool expected)
{
var p = new ColumbiaProduct { Sku = sku, Name = name };
Assert.Equal(expected, ColumbiaCatalogMapper.IsExcludedProduct(p));
}
// ── Pricing ───────────────────────────────────────────────────────────
[Fact]
public void ParseBasePrice_SimpleProduct_UsesTopLevelPrice()
{
var p = new ColumbiaProduct { Price = "18.85", RegularPrice = "18.85" };
Assert.Equal(18.85m, ColumbiaCatalogMapper.ParseBasePrice(p));
}
[Fact]
public void ParseBasePrice_VariableProductWithZeroRegular_FallsBackToPriceThenVariants()
{
// Variable parent: price carries the lead variant, regular_price is "0".
var p = new ColumbiaProduct
{
Price = "18.85",
RegularPrice = "0",
VariationPricing = new List<ColumbiaVariationPricing>
{
new() { Sku = "X-B", Price = "18.85" },
new() { Sku = "X-P", Price = "18.85" },
},
};
Assert.Equal(18.85m, ColumbiaCatalogMapper.ParseBasePrice(p));
}
[Fact]
public void BuildPriceTiersJson_VariableProduct_SerializesVariationPricing()
{
var p = new ColumbiaProduct
{
VariationPricing = new List<ColumbiaVariationPricing>
{
new() { Sku = "X-B", Price = "18.85" },
},
};
var json = ColumbiaCatalogMapper.BuildPriceTiersJson(p);
Assert.NotNull(json);
Assert.Contains("X-B", json);
}
[Fact]
public void BuildPriceTiersJson_EmptyTieredPricingArray_ReturnsNull()
{
// Variable products carry tiered_pricing as an empty array.
var p = new ColumbiaProduct { TieredPricing = JsonDocument.Parse("[]").RootElement };
Assert.Null(ColumbiaCatalogMapper.BuildPriceTiersJson(p));
}
// ── Chemistry, color, HTML ────────────────────────────────────────────
[Theory]
[InlineData("Polyester/TGIC", "Polyester/TGIC")]
[InlineData("Polyester TGIC", "Polyester/TGIC")]
[InlineData("TGIC Polyester", "Polyester/TGIC")]
[InlineData("TGIC", "TGIC")]
[InlineData("", null)]
public void NormalizeChemistry_CollapsesPolyesterTgicVariants(string input, string? expected)
{
Assert.Equal(expected, ColumbiaCatalogMapper.NormalizeChemistry(input));
}
[Fact]
public void BuildColorFamilies_JoinsColorGroupNames()
{
var p = new ColumbiaProduct
{
PaColorGroup = { new ColumbiaNamed { Name = "Blue" }, new ColumbiaNamed { Name = "Green" } },
};
Assert.Equal("Blue,Green", ColumbiaCatalogMapper.BuildColorFamilies(p));
}
[Fact]
public void BuildColorFamilies_FallsBackToColorGroupAttribute()
{
var p = new ColumbiaProduct
{
Attributes =
{
new ColumbiaAttribute
{
Name = "Color Group",
Options = { new ColumbiaNamed { Name = "Black" } },
},
},
};
Assert.Equal("Black", ColumbiaCatalogMapper.BuildColorFamilies(p));
}
[Fact]
public void StripHtml_RemovesTagsEntitiesAndCollapsesWhitespace()
{
var html = "<strong>Blue Wrinkle</strong>\r\n\r\nThis is a vibrant &amp; bright coating.";
var text = ColumbiaCatalogMapper.StripHtml(html);
Assert.Equal("Blue Wrinkle This is a vibrant & bright coating.", text);
}
[Fact]
public void DetectRequiresClearCoat_ExplicitRequirement_IsTrue()
{
var p = new ColumbiaProduct
{
Description = "This powder requires a clear coat to activate the effect.",
};
Assert.True(ColumbiaCatalogMapper.DetectRequiresClearCoat(p));
}
[Fact]
public void DetectRequiresClearCoat_IllusionLine_IsTrue()
{
var p = new ColumbiaProduct { Name = "Illusion Cherry", Description = "A translucent red." };
Assert.True(ColumbiaCatalogMapper.DetectRequiresClearCoat(p));
}
[Fact]
public void DetectRequiresClearCoat_PartialCureSchedule_IsTrue()
{
var p = new ColumbiaProduct
{
Name = "Some Base",
CureSchedule = "Partial Cure: 15 min total time in oven preheated to 400°F. Then apply clear.",
};
Assert.True(ColumbiaCatalogMapper.DetectRequiresClearCoat(p));
}
[Fact]
public void DetectRequiresClearCoat_CasualMention_IsFalse()
{
// The over-flagging case: a passing mention is not a requirement.
var p = new ColumbiaProduct
{
Name = "Gloss Black",
Description = "Durable gloss black. Apply a clear coat for added protection if desired.",
};
Assert.False(ColumbiaCatalogMapper.DetectRequiresClearCoat(p));
}
// ── End-to-end mapping invariants ─────────────────────────────────────
// ── Tolerant image deserialization (WordPress returns [] / false when empty) ──
private static readonly JsonSerializerOptions ClientJsonOptions = new()
{
PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower,
PropertyNameCaseInsensitive = true,
Converters = { new ColumbiaImageJsonConverter() },
};
[Fact]
public void Deserialize_FeaturedImageEmptyArray_YieldsNullImage()
{
// WordPress returns featured_image: [] for products with no image.
var json = """{ "sku": "X1", "name": "No Image", "featured_image": [] }""";
var product = JsonSerializer.Deserialize<ColumbiaProduct>(json, ClientJsonOptions);
Assert.NotNull(product);
Assert.Null(product!.FeaturedImage);
Assert.Null(ColumbiaCatalogMapper.Map(product).ImageUrl);
}
[Fact]
public void Deserialize_FeaturedImageObject_ParsesSrc()
{
var json = """{ "sku": "X1", "name": "Has Image", "featured_image": { "id": 5, "src": "https://x/img.png", "name": "i", "alt": "" } }""";
var product = JsonSerializer.Deserialize<ColumbiaProduct>(json, ClientJsonOptions);
Assert.Equal("https://x/img.png", product!.FeaturedImage!.Src);
Assert.Equal("https://x/img.png", ColumbiaCatalogMapper.Map(product).ImageUrl);
}
[Fact]
public void Map_AlwaysStampsSource_AndLeavesEnrichmentFieldsNull()
{
var p = new ColumbiaProduct { Sku = "X5004124", Name = "Blue Wrinkle", Price = "18.85" };
var item = ColumbiaCatalogMapper.Map(p);
Assert.Equal("Columbia Coatings API", item.Source);
// Enrichment fields are not in the feed and must stay null for lazy TDS/AI enrichment.
Assert.Null(item.SpecificGravity);
Assert.Null(item.CoverageSqFtPerLb);
Assert.Null(item.TransferEfficiency);
}
}
@@ -0,0 +1,222 @@
using System.Security.Claims;
using Microsoft.AspNetCore.Http;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging;
using Moq;
using PowderCoating.Core.Entities;
using PowderCoating.Infrastructure.Data;
using PowderCoating.Infrastructure.Repositories;
using PowderCoating.Infrastructure.Services;
namespace PowderCoating.UnitTests;
/// <summary>
/// Verifies that catalog sync propagation updates a linked inventory item's quoting reference price
/// and product data, while never touching the tenant-owned cost basis, quantity, notes, or image.
/// </summary>
public class PowderCatalogPropagationTests
{
[Fact]
public async Task Propagate_UpdatesReferencePriceAndSpecs_ButNotCostQuantityNotesOrImage()
{
await using var context = CreateContext();
var catalog = new PowderCatalogItem
{
VendorName = "Columbia Coatings",
Sku = "CS1693053",
ColorName = "Joker Jewel",
Source = "Columbia Coatings API",
UnitPrice = 28m, // new catalog price
SdsUrl = "https://cc/sds.pdf",
TdsUrl = "https://cc/tds.pdf",
CureTemperatureF = 400m,
CureTimeMinutes = 10,
ColorFamilies = "Green,Purple",
};
context.PowderCatalogItems.Add(catalog);
await context.SaveChangesAsync();
var inv = new InventoryItem
{
CompanyId = 1,
SKU = "POWD-2606-0001",
Name = "Joker Jewel",
PowderCatalogItemId = catalog.Id,
UnitCost = 20m, // what they actually paid
AverageCost = 20m,
LastPurchasePrice = 20m,
QuantityOnHand = 5m,
Notes = "keep my note",
ImageUrl = "my-own-photo.jpg",
CatalogReferencePrice = null, // not yet set
};
context.InventoryItems.Add(inv);
await context.SaveChangesAsync();
var service = new PowderCatalogUpsertService(
new UnitOfWork(context),
Mock.Of<ILogger<PowderCatalogUpsertService>>());
var updated = await service.PropagateToLinkedInventoryAsync();
Assert.Equal(1, updated);
var refreshed = await context.InventoryItems.FindAsync(inv.Id);
Assert.NotNull(refreshed);
// Quoting reference price + product data refreshed from the catalog.
Assert.Equal(28m, refreshed!.CatalogReferencePrice);
Assert.NotNull(refreshed.CatalogPriceUpdatedAt);
Assert.Equal("https://cc/sds.pdf", refreshed.SdsUrl);
Assert.Equal("https://cc/tds.pdf", refreshed.TdsUrl);
Assert.Equal(400m, refreshed.CureTemperatureF);
Assert.Equal("Green,Purple", refreshed.ColorFamilies);
// Tenant-owned fields untouched.
Assert.Equal(20m, refreshed.UnitCost);
Assert.Equal(20m, refreshed.AverageCost);
Assert.Equal(20m, refreshed.LastPurchasePrice);
Assert.Equal(5m, refreshed.QuantityOnHand);
Assert.Equal("keep my note", refreshed.Notes);
Assert.Equal("my-own-photo.jpg", refreshed.ImageUrl);
}
[Fact]
public async Task Propagate_DoesNotSetReferencePrice_WhenCatalogPriceIsZero()
{
await using var context = CreateContext();
var catalog = new PowderCatalogItem
{
VendorName = "Columbia Coatings",
Sku = "X1",
ColorName = "No Price",
Source = "Columbia Coatings API",
UnitPrice = 0m, // unknown price — must not wipe quoting with $0
};
context.PowderCatalogItems.Add(catalog);
await context.SaveChangesAsync();
var inv = new InventoryItem
{
CompanyId = 1,
SKU = "POWD-2606-0002",
Name = "No Price",
PowderCatalogItemId = catalog.Id,
UnitCost = 15m,
CatalogReferencePrice = null,
};
context.InventoryItems.Add(inv);
await context.SaveChangesAsync();
var service = new PowderCatalogUpsertService(
new UnitOfWork(context),
Mock.Of<ILogger<PowderCatalogUpsertService>>());
await service.PropagateToLinkedInventoryAsync();
var refreshed = await context.InventoryItems.FindAsync(inv.Id);
Assert.Null(refreshed!.CatalogReferencePrice); // stays null -> quoting falls back to UnitCost
}
[Fact]
public async Task Propagate_LinksUnlinkedItem_ByManufacturerAndPartNumber()
{
await using var context = CreateContext();
var catalog = new PowderCatalogItem
{
VendorName = "Columbia Coatings",
Sku = "CS1693053",
ColorName = "Joker Jewel",
Source = "Columbia Coatings API",
UnitPrice = 28m,
};
context.PowderCatalogItems.Add(catalog);
await context.SaveChangesAsync();
var inv = new InventoryItem
{
CompanyId = 1,
SKU = "POWD-2606-0009",
Name = "Joker Jewel",
Manufacturer = "Columbia Coatings",
ManufacturerPartNumber = "CS1693053", // matches catalog SKU
PowderCatalogItemId = null, // not linked yet (legacy item)
UnitCost = 20m,
};
context.InventoryItems.Add(inv);
await context.SaveChangesAsync();
var service = new PowderCatalogUpsertService(
new UnitOfWork(context),
Mock.Of<ILogger<PowderCatalogUpsertService>>());
await service.PropagateToLinkedInventoryAsync();
var refreshed = await context.InventoryItems.FindAsync(inv.Id);
Assert.Equal(catalog.Id, refreshed!.PowderCatalogItemId); // self-healed link
Assert.Equal(28m, refreshed.CatalogReferencePrice); // and got the price
Assert.Equal(20m, refreshed.UnitCost); // cost untouched
}
[Fact]
public async Task Propagate_DoesNotLink_WhenPartNumberDoesNotMatch()
{
await using var context = CreateContext();
context.PowderCatalogItems.Add(new PowderCatalogItem
{
VendorName = "Columbia Coatings",
Sku = "CS1693053",
ColorName = "Joker Jewel",
UnitPrice = 28m,
});
await context.SaveChangesAsync();
var inv = new InventoryItem
{
CompanyId = 1,
SKU = "POWD-2606-0010",
Name = "Something Else",
Manufacturer = "Columbia Coatings",
ManufacturerPartNumber = "NOPE-999", // no catalog match
PowderCatalogItemId = null,
};
context.InventoryItems.Add(inv);
await context.SaveChangesAsync();
var service = new PowderCatalogUpsertService(
new UnitOfWork(context),
Mock.Of<ILogger<PowderCatalogUpsertService>>());
await service.PropagateToLinkedInventoryAsync();
var refreshed = await context.InventoryItems.FindAsync(inv.Id);
Assert.Null(refreshed!.PowderCatalogItemId); // stays unlinked
}
private static ApplicationDbContext CreateContext()
{
var options = new DbContextOptionsBuilder<ApplicationDbContext>()
.UseInMemoryDatabase(Guid.NewGuid().ToString())
.Options;
var identity = new ClaimsIdentity([new Claim(ClaimTypes.Role, "SuperAdmin")], "Test");
var principal = new ClaimsPrincipal(identity);
byte[]? noBytes = null;
var sessionMock = new Mock<ISession>();
sessionMock.Setup(s => s.TryGetValue(It.IsAny<string>(), out noBytes)).Returns(false);
var httpContextMock = new Mock<HttpContext>();
httpContextMock.SetupGet(c => c.User).Returns(principal);
httpContextMock.SetupGet(c => c.Session).Returns(sessionMock.Object);
var accessor = new Mock<IHttpContextAccessor>();
accessor.SetupGet(a => a.HttpContext).Returns(httpContextMock.Object);
return new ApplicationDbContext(options, accessor.Object, null!);
}
}