feat(seo): integrate analytics worker and real HTTP PostgreSQL acceptance
This commit is contained in:
1 parent
5750e1589e
commit
312fcc2acc
19 files changed
+1120
No files matched your search
@@ -0,0 +1,26 @@
|
||||
# SEO tracking worker — local, not deployed
|
||||
|
||||
Run tests: `python3 -m unittest discover -v`
|
||||
|
||||
Read-only actual Google collection:
|
||||
`python3 seo_performance_tracker.py --probe-url https://canina.ir/blog/dog-black-seed-nigella-sativa --state article-probe.sqlite --receipt live-article-receipt.json`
|
||||
|
||||
Read all research findings with strict pagination: `python3 seo_performance_tracker.py --backend https://HOST --state tracker.sqlite --receipt receipt.json`. Supply `CANINA_ADMIN_TOKEN` through the environment, never command arguments/logs. CLI performs no remote mutations. Dependencies: Python 3, requests, google-auth. Credentials stay in their original path.
|
||||
|
||||
`deliver_snapshots(store, backend)` is an explicit opt-in library operation, exercised against both a local HTTP fixture and the actual Nest module over TCP with disposable PostgreSQL (see `VERIFICATION.md`). It posts the sibling v2 ingestion contract (contract.md) and retains immutable payload for retries. Local keys include finding, date period, feedbackVersion. It rereads feedbackVersion before delivery, but this is not an atomic guard; snapshot metadata records the observed revision and snapshots never execute article directives. Actual article creation must atomically claim backend `/action` with expectedVersion and feedbackVersion; this worker does not publish articles. SourceMetadata.worker retains Clarity, query rows, provenance and mandatory directive.
|
||||
|
||||
`deliver_alerts(store, session, endpoint)` is an opt-in structured department receiver adapter. Pending events remain durable in SQLite, deduplicated per source/error/finding/day; successful receiver HTTP acknowledgement marks delivery. Receiver must itself persist and deduplicate idempotencyKey. No actual Telegram/department receiver configured or tested. No background daemon/cron installed. Automatic retry is not scheduled; rerun explicit delivery to retry.
|
||||
|
||||
Measurement: inclusive 14 date labels ending UTC today minus 3 days, GSC final; GA4 timezone metadata retained. For the verified property GA4 returned America/Los_Angeles, aligning GSC date labels, but not proving session-level cross-source attribution. Page metrics queried separately from anonymized/top-row query data; never sum query rows to invent full page metrics. No rows = null/no_data, not zero. Scroll unavailable absent instrumentation. Query response may omit anonymized queries. GA4 thresholding metadata retained; no automated scaling/pruning policy implemented without approved comparative evidence.
|
||||
|
||||
## Verified read-only outcome (2026-10-28)
|
||||
Actual REST requests succeeded using AuthorizedSession. For 2026-10-12 through 2026-10-25, sampled existing article `/blog/dog-black-seed-nigella-sativa`: GSC 5 clicks, 32 impressions, position 3.4375; GA4 2 active users, 33 total engagement seconds, average 16.5 seconds/user, bounce 0.5. GSC query rows empty despite nonzero page metrics (not contradictory; query coverage is limited). Two runs produced one immutable local snapshot and two pending unavailable-source alerts. Receipts contain actual request results, not authenticated-session-construction claims. Discovery fetched a single sample page and is explicitly not a site-wide audit.
|
||||
|
||||
## Blockers / boundaries
|
||||
- Clarity frontend project wiring exists; reporting credentials/access not verified. No Clarity data fabricated.
|
||||
- Repo Order/OrderItem has product linkage but no observed article landing/session/order identity chain; related product sales are correlation, not article/keyword causal attribution. Commerce metrics remain null.
|
||||
- Authenticated list/ingestion now exercised against the actual Nest SEO module and real disposable PostgreSQL with a test-only JWT strategy and the shared guards. This does not verify production identity lookup or API-key authentication. Real department/Telegram receipt remains unverified; production writes forbidden.
|
||||
- Mandatory CEO directives carried with revision; existing article publishers are not integrated by this worker.
|
||||
- Integrated into the authorized develop worktree under `scripts/seo-performance/`. No push/Stage/deploy/cron. See `VERIFICATION.md` for exact local acceptance and limitations.
|
||||
- Historical spec zero defaults/threshold recommendations were unsafe and intentionally not restored. Local snapshot same-key remeasurement is immutable; a new period/revision creates a new record. Correcting an earlier snapshot requires an explicit future revision design.
|
||||
- See `VERIFICATION.md` for source baseline and graph provenance. Original graph is SHA-unbound navigation, not completeness proof.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Worker integration acceptance — 2026-10-28
|
||||
|
||||
Source baseline: develop `df5584d21461f68eb05ffb59ae79ddde4acc622e`, including backend commits 6102556 and 39f5b70. Existing graph-cache timestamp, .specify/ and evidence/ changes were not owned and are excluded. Sibling consumer work was excluded.
|
||||
|
||||
Graph navigation: copied graph.json to `/opt/data/cache/scratch/seo-http-impact`, queried `SeoTrackingController SeoTrackingService authentication snapshots worker`. Original SHA256 6a73473cee9e207543fdeeb668bea544f766200a1b16b8a3a9ce18d4e42e51af; source binding unknown. Graph is navigation only; directly inspected module/controller/service, PrismaService, tests and worker contract at baseline.
|
||||
|
||||
## Executed
|
||||
- Python `python3 -m unittest discover -v`: 17 tests pass, exit 0.
|
||||
- Backend `SEO_TEST_DATABASE_URL=postgresql://seo_test@127.0.0.1:55439/postgres npm test -- --runInBand seo-tracking`: 4 suites, 16 tests pass, exit 0. Receipt in evidence/real-http-receipt.json and focused output in evidence/backend-focused.txt.
|
||||
- `npm run build`: exit 0.
|
||||
- Full backend `npm test -- --runInBand` without integration env: 94 suites pass, 1 fails, 4 skip; 815 tests pass, 1 fails, 8 skip. Failure is existing WikiService pagination assertion, previously reproduced on clean baseline by backend owner; see backend/src/seo-tracking/VERIFICATION.md. Full output in evidence/backend-full.txt. This is NOT full-suite green.
|
||||
- Spec Kit 1.2.3 prerequisite script with explicit feature directory, --require-spec --require-tasks --include-tasks: exit 0. Artifact presence only, not full acceptance.
|
||||
|
||||
New integration test imports the actual SeoTrackingModule, replaces only PrismaService with a real PrismaClient pointed at disposable PostgreSQL, uses a local Passport JWT strategy with actual shared JwtAuthGuard/RolesGuard, and binds loopback ephemeral TCP. Python calls real requests, creates finding, patches feedback (version 1), rejects stale feedback action (409), links correct action (version 2), reads worker findings, makes actual read-only Google requests, persists SQLite state, closes/reopens state, delivers snapshot through real HTTP, GETs database history, retries identical POST (same ID), rejects changed payload (409), and verifies one history row. Missing/invalid JWT and customer role are denied. Fixture blog/finding and cascading snapshots are cleaned up. This is test-auth, NOT production user lookup/Redis/API-key acceptance.
|
||||
|
||||
Actual Google result for existing published blackseed article, 2026-10-12..25: HTTP 200 GSC 5 clicks / 32 impressions / position 3.4375; query rows empty (limited coverage). HTTP 200 GA4 2 active users / 33 engagement seconds / 16.5 seconds per user / bounce 0.5; timezone America/Los_Angeles. Scroll, commerce and Clarity remain null/unavailable. No causal sales or automated scaling claim.
|
||||
|
||||
First integration run failed because pg_ctl defaulted to port 5432, not the allowed 55439. Restarted this known disposable cluster with explicit loopback port 55439 and scratch Unix socket; rerun passed. This was test setup failure, NOT behavioral RED or evidence of a contract bug. No worker/backend contract mismatch was found, so no production implementation fix or claimed new RED/GREEN cycle. New integration assertions are acceptance coverage of existing code.
|
||||
|
||||
## Reproduce
|
||||
Provision a disposable PostgreSQL 16 schema from backend/prisma/schema.prisma (never production) at the exact test URL. Then from backend:
|
||||
|
||||
`SEO_TEST_DATABASE_URL=postgresql://seo_test@127.0.0.1:55439/postgres SEO_HTTP_RECEIPT=/absolute/scratch/receipt.json npm test -- --runInBand seo-tracking.worker-http.spec.ts`
|
||||
|
||||
Requires Python requests/google-auth and existing read-only Google credential path used by worker; never copy credentials into Git. This opt-in test makes Google reads and only local test-database writes. No cron or deployment exists.
|
||||
|
||||
## Still open
|
||||
Actual publisher consumers/finalization enforcement, many-actions/goals/history requirements, production identity/API-key, browser/full frontend matrix, real department alert receipt, Clarity reporting, event instrumentation and causal commerce attribution. Full release acceptance remains open; no push/CI/Stage/deployment/production writes authorized or performed. Broader spec tasks intentionally remain unchecked rather than conflating partial evidence with closure.
|
||||
@@ -0,0 +1,7 @@
|
||||
# SEO tracking contract v2 (local implementation)
|
||||
Base `/api/admin/seo-tracking`, existing JWT or API key authentication mandatory, admin role (API keys require admin scope). No new unauthenticated worker path.
|
||||
GET list page integer 1..100000, limit 1..100, status enum, sortBy createdAt|updatedAt|title|status, sortOrder asc|desc; returns data and standard meta. GET /:uuid returns finding including version, feedbackVersion, blog, snapshots.
|
||||
POST finding: title,hypothesis,targetKeywords[],primaryKeyword,expectedOutcome required; optional blogId UUID, status, goalMetric (metric name), goalTarget nonnegative, actionId external reference. PATCH /:id feedback: {feedback:string, expectedVersion:integer>=0} required; atomic claim advances version and feedbackVersion; stale 409; return updated finding. Clearing feedback uses empty string and advances version.
|
||||
PATCH /:id/action: {expectedVersion, feedbackVersion, blogId?, actionId?, status?}; atomic finding version AND feedbackVersion claim. Feedback present requires acknowledgement of exact feedbackVersion. This protects this endpoint; existing blog publishers are NOT automatically integrated and must consume this contract before claiming enforcement.
|
||||
POST /:id/snapshots: ingestionKey string <=128, periodStart/periodEnd ISO timestamps (start<end), sourceMetadata object with gsc,ga4,commerce each {status: available|unavailable|error, source:string, fetchedAt:ISO timestamp, error?:string}; optional metric fields. Metrics absent/null remain null, actual zero remains zero. All non-available source metrics MUST be null/omitted. Available sources require all their metrics except ga4ScrollDepthRate (optional if uninstrumented). Counts integer >=0, rates [0,1], other metrics >=0 finite. Key unique per finding: identical retry returns existing; changed payload same key returns 409. No revenue inferred from aggregate orders. No recommendations from incomplete data; ingestion accepts optional recommendation only with all sources available.
|
||||
Legacy snapshots with null provenance are unverified historical data, not measured zero. Legacy findings require normalized searchText backfill before normalized search can be complete. No live migration/scheduling/deployment authorized.
|
||||
@@ -0,0 +1,8 @@
|
||||
npm notice run backend@0.0.1 test
|
||||
npm notice run jest --runInBand seo-tracking
|
||||
|
||||
Test Suites: 4 passed, 4 total
|
||||
Tests: 16 passed, 16 total
|
||||
Snapshots: 0 total
|
||||
Time: 10.064 s
|
||||
Ran all test suites matching seo-tracking.
|
||||
@@ -0,0 +1,165 @@
|
||||
npm notice run backend@0.0.1 test
|
||||
npm notice run jest --runInBand
|
||||
FAIL src/wiki/wiki.service.spec.ts
|
||||
● WikiService › findAll › should paginate and filter terms with fallback to default sortBy
|
||||
|
||||
expect(jest.fn()).toHaveBeenCalledWith(...expected)
|
||||
|
||||
- Expected
|
||||
+ Received
|
||||
|
||||
@@ -1,21 +1,39 @@
|
||||
Object {
|
||||
"orderBy": Object {
|
||||
"term": "desc",
|
||||
},
|
||||
- "skip": 0,
|
||||
- "take": 10,
|
||||
+ "skip": undefined,
|
||||
+ "take": 100,
|
||||
"where": Object {
|
||||
"OR": Array [
|
||||
Object {
|
||||
"term": Object {
|
||||
"contains": "مفصل",
|
||||
"mode": "insensitive",
|
||||
},
|
||||
},
|
||||
Object {
|
||||
+ "key": Object {
|
||||
+ "contains": "مفصل",
|
||||
+ "mode": "insensitive",
|
||||
+ },
|
||||
+ },
|
||||
+ Object {
|
||||
"definition": Object {
|
||||
+ "contains": "مفصل",
|
||||
+ "mode": "insensitive",
|
||||
+ },
|
||||
+ },
|
||||
+ Object {
|
||||
+ "term": Object {
|
||||
+ "contains": "مفصل",
|
||||
+ "mode": "insensitive",
|
||||
+ },
|
||||
+ },
|
||||
+ Object {
|
||||
+ "key": Object {
|
||||
"contains": "مفصل",
|
||||
"mode": "insensitive",
|
||||
},
|
||||
},
|
||||
],,
|
||||
|
||||
Number of calls: 1
|
||||
|
||||
56 | });
|
||||
57 |
|
||||
> 58 | expect(mockPrisma.scientificTerm.findMany).toHaveBeenCalledWith({
|
||||
| ^
|
||||
59 | where: {
|
||||
60 | OR: [
|
||||
61 | { term: { contains: "مفصل", mode: "insensitive" } },
|
||||
|
||||
at Object.<anonymous> (wiki/wiki.service.spec.ts:58:50)
|
||||
|
||||
[Nest] 3581768 - 10/28/2026, 7:18:58 AM ERROR [SslService] Failed to parse SSL certificate
|
||||
[Nest] 3581768 - 10/28/2026, 7:18:58 AM ERROR [SslService] Error: error:0480006C:PEM routines::no start line
|
||||
at new X509Certificate (node:internal/crypto/x509:121:21)
|
||||
at SslService.getStatus (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/src/admin/ssl.service.ts:104:20)
|
||||
at Object.<anonymous> (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/src/admin/ssl.service.spec.ts:179:30)
|
||||
at Promise.finally.completed (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-circus/build/jestAdapterInit.js:1561:28)
|
||||
at new Promise (<anonymous>)
|
||||
at callAsyncCircusFn (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-circus/build/jestAdapterInit.js:1501:10)
|
||||
at _callCircusTest (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-circus/build/jestAdapterInit.js:1011:40)
|
||||
at _runTest (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-circus/build/jestAdapterInit.js:951:3)
|
||||
at /opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-circus/build/jestAdapterInit.js:853:7
|
||||
at _runTestsForDescribeBlock (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-circus/build/jestAdapterInit.js:866:11)
|
||||
at _runTestsForDescribeBlock (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-circus/build/jestAdapterInit.js:861:11)
|
||||
at _runTestsForDescribeBlock (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-circus/build/jestAdapterInit.js:861:11)
|
||||
at run (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-circus/build/jestAdapterInit.js:765:3)
|
||||
at runAndTransformResultsToJestFormat (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-circus/build/jestAdapterInit.js:1993:21)
|
||||
at jestAdapter (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-circus/build/runner.js:111:19)
|
||||
at runTestInternal (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-runner/build/index.js:276:16)
|
||||
at runTest (/opt/data/content-maintenance/engine/worktree-concurrency-tdd/backend/node_modules/jest-runner/build/index.js:344:7) {
|
||||
library: 'PEM routines',
|
||||
reason: 'no start line',
|
||||
code: 'ERR_OSSL_PEM_NO_START_LINE'
|
||||
}
|
||||
[Nest] 3581768 - 10/28/2026, 7:19:05 AM ERROR [ZibalEBankService] [Zibal EBank API] Error requesting /ebank/v1/account/list/: transport failed
|
||||
[Nest] 3581768 - 10/28/2026, 7:19:05 AM ERROR [ZibalEBankService] [Zibal EBank API] Error requesting /ebank/v1/account/list/: transport failed
|
||||
[Nest] 3581768 - 10/28/2026, 7:19:20 AM ERROR [ZibalEBankService] [Zibal EBank API] Error requesting /ebank/v1/account/list/: aborted
|
||||
[Nest] 3581768 - 10/28/2026, 7:19:05 AM ERROR [ZibalEBankService] [Zibal EBank API] Error requesting /ebank/v1/account/list/: Zibal EBank HTTP 503
|
||||
[Nest] 3581768 - 10/28/2026, 7:19:17 AM ERROR [PaymentService] Security Alert: Amount mismatch on trackId=tr-1! Zibal reported 1000000 Rials, but expected 5000000 Rials in DB.
|
||||
[Nest] 3581768 - 10/28/2026, 7:19:27 AM WARN [BlogsService] Cache revalidation failed for updated blog [post]: cache down
|
||||
|
||||
Summary of all failing tests
|
||||
FAIL wiki/wiki.service.spec.ts
|
||||
● WikiService › findAll › should paginate and filter terms with fallback to default sortBy
|
||||
|
||||
expect(jest.fn()).toHaveBeenCalledWith(...expected)
|
||||
|
||||
- Expected
|
||||
+ Received
|
||||
|
||||
@@ -1,21 +1,39 @@
|
||||
Object {
|
||||
"orderBy": Object {
|
||||
"term": "desc",
|
||||
},
|
||||
- "skip": 0,
|
||||
- "take": 10,
|
||||
+ "skip": undefined,
|
||||
+ "take": 100,
|
||||
"where": Object {
|
||||
"OR": Array [
|
||||
Object {
|
||||
"term": Object {
|
||||
"contains": "مفصل",
|
||||
"mode": "insensitive",
|
||||
},
|
||||
},
|
||||
Object {
|
||||
+ "key": Object {
|
||||
+ "contains": "مفصل",
|
||||
+ "mode": "insensitive",
|
||||
+ },
|
||||
+ },
|
||||
+ Object {
|
||||
"definition": Object {
|
||||
+ "contains": "مفصل",
|
||||
+ "mode": "insensitive",
|
||||
+ },
|
||||
+ },
|
||||
+ Object {
|
||||
+ "term": Object {
|
||||
+ "contains": "مفصل",
|
||||
+ "mode": "insensitive",
|
||||
+ },
|
||||
+ },
|
||||
+ Object {
|
||||
+ "key": Object {
|
||||
"contains": "مفصل",
|
||||
"mode": "insensitive",
|
||||
},
|
||||
},
|
||||
],,
|
||||
|
||||
Number of calls: 1
|
||||
|
||||
56 | });
|
||||
57 |
|
||||
> 58 | expect(mockPrisma.scientificTerm.findMany).toHaveBeenCalledWith({
|
||||
| ^
|
||||
59 | where: {
|
||||
60 | OR: [
|
||||
61 | { term: { contains: "مفصل", mode: "insensitive" } },
|
||||
|
||||
at Object.<anonymous> (wiki/wiki.service.spec.ts:58:50)
|
||||
|
||||
|
||||
Test Suites: 1 failed, 4 skipped, 94 passed, 95 of 99 total
|
||||
Tests: 1 failed, 8 skipped, 815 passed, 824 total
|
||||
Snapshots: 0 total
|
||||
Time: 44.533 s, estimated 47 s
|
||||
Ran all test suites.
|
||||
@@ -0,0 +1,254 @@
|
||||
{
|
||||
"observedAt": "2026-10-28T07:18:43.957972+00:00",
|
||||
"backend": "disposable local Nest module + real PostgreSQL",
|
||||
"auth": "test-only JWT strategy with real JwtAuthGuard/RolesGuard; NOT production identity/API-key",
|
||||
"checks": {
|
||||
"missing_invalid_nonadmin_denied": true,
|
||||
"feedback_stale_action_and_correct_link": true,
|
||||
"worker_snapshot_history_network_retry_conflict": true
|
||||
},
|
||||
"google": {
|
||||
"findingId": "c89ce29f-e9f7-40ab-a7bb-03a2286219aa",
|
||||
"articleUrl": "https://canina.ir/blog/dog-black-seed-nigella-sativa",
|
||||
"periodStart": "2026-10-12",
|
||||
"periodEnd": "2026-10-25",
|
||||
"feedbackRevision": 1,
|
||||
"sources": {
|
||||
"gsc": {
|
||||
"status": "available",
|
||||
"metrics": {
|
||||
"clicks": 5,
|
||||
"impressions": 32,
|
||||
"position": 3.4375
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"keys": [
|
||||
"https://canina.ir/blog/dog-black-seed-nigella-sativa"
|
||||
],
|
||||
"clicks": 5,
|
||||
"impressions": 32,
|
||||
"ctr": 0.15625,
|
||||
"position": 3.4375
|
||||
}
|
||||
],
|
||||
"periodStart": "2026-10-12",
|
||||
"periodEnd": "2026-10-25",
|
||||
"observedAt": "2026-10-28T07:18:42.716059+00:00",
|
||||
"metadata": {},
|
||||
"coverage": "GSC top rows; anonymized queries omitted",
|
||||
"source": "GSC",
|
||||
"httpStatus": 200
|
||||
},
|
||||
"gscQueries": {
|
||||
"status": "no_data",
|
||||
"metrics": null,
|
||||
"rows": [],
|
||||
"periodStart": "2026-10-12",
|
||||
"periodEnd": "2026-10-25",
|
||||
"observedAt": "2026-10-28T07:18:42.979366+00:00",
|
||||
"metadata": {},
|
||||
"coverage": "GSC top rows; anonymized queries omitted",
|
||||
"source": "GSC",
|
||||
"httpStatus": 200
|
||||
},
|
||||
"ga4": {
|
||||
"status": "available",
|
||||
"metrics": {
|
||||
"activeUsers": 2,
|
||||
"userEngagementDuration": 33.0,
|
||||
"avgEngagementSec": 16.5,
|
||||
"bounceRate": 0.5,
|
||||
"scrollDepthRate": null
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"dimensionValues": [
|
||||
{
|
||||
"value": "https://canina.ir/blog/dog-black-seed-nigella-sativa"
|
||||
}
|
||||
],
|
||||
"metricValues": [
|
||||
{
|
||||
"value": "2"
|
||||
},
|
||||
{
|
||||
"value": "33"
|
||||
},
|
||||
{
|
||||
"value": "0.5"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"periodStart": "2026-10-12",
|
||||
"periodEnd": "2026-10-25",
|
||||
"observedAt": "2026-10-28T07:18:43.791790+00:00",
|
||||
"metadata": {
|
||||
"currencyCode": "USD",
|
||||
"timeZone": "America/Los_Angeles"
|
||||
},
|
||||
"coverage": "GA4 reporting rows; inspect metadata for thresholding",
|
||||
"source": "GA4",
|
||||
"httpStatus": 200
|
||||
},
|
||||
"clarity": {
|
||||
"status": "unavailable",
|
||||
"metrics": null,
|
||||
"error": "reporting_credentials_not_configured",
|
||||
"observedAt": "2026-10-28T07:18:43.791896+00:00"
|
||||
},
|
||||
"commerce": {
|
||||
"status": "unavailable",
|
||||
"metrics": null,
|
||||
"error": "article_to_order_linkage_not_verified",
|
||||
"observedAt": "2026-10-28T07:18:43.791906+00:00"
|
||||
}
|
||||
},
|
||||
"decision": {
|
||||
"action": "INSUFFICIENT_EVIDENCE",
|
||||
"reason": "No approved comparative decision policy; unavailable signals cannot justify scaling or pruning. Correlation is not keyword-caused sales.",
|
||||
"mandatoryDirective": "Measure only; do not publish",
|
||||
"feedbackRevision": 1
|
||||
},
|
||||
"observedAt": "2026-10-28T07:18:43.791921+00:00",
|
||||
"periodAlignment": "same date labels; GSC Pacific and GA4 property timezone may differ; not session-level alignment"
|
||||
},
|
||||
"persistedSnapshot": {
|
||||
"id": "26dac670-0b45-4450-bbec-638446732554",
|
||||
"findingId": "c89ce29f-e9f7-40ab-a7bb-03a2286219aa",
|
||||
"snapshotDate": "2026-10-28T07:18:43.874Z",
|
||||
"periodStart": "2026-10-12T00:00:00.000Z",
|
||||
"periodEnd": "2026-10-26T00:00:00.000Z",
|
||||
"sourceMetadata": {
|
||||
"ga4": {
|
||||
"source": "ga4",
|
||||
"status": "available",
|
||||
"fetchedAt": "2026-10-28T07:18:43.791790+00:00"
|
||||
},
|
||||
"gsc": {
|
||||
"source": "gsc",
|
||||
"status": "available",
|
||||
"fetchedAt": "2026-10-28T07:18:42.716059+00:00"
|
||||
},
|
||||
"worker": {
|
||||
"sources": {
|
||||
"ga4": {
|
||||
"rows": [
|
||||
{
|
||||
"metricValues": [
|
||||
{
|
||||
"value": "2"
|
||||
},
|
||||
{
|
||||
"value": "33"
|
||||
},
|
||||
{
|
||||
"value": "0.5"
|
||||
}
|
||||
],
|
||||
"dimensionValues": [
|
||||
{
|
||||
"value": "https://canina.ir/blog/dog-black-seed-nigella-sativa"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": "GA4",
|
||||
"status": "available",
|
||||
"metrics": {
|
||||
"bounceRate": 0.5,
|
||||
"activeUsers": 2,
|
||||
"scrollDepthRate": null,
|
||||
"avgEngagementSec": 16.5,
|
||||
"userEngagementDuration": 33
|
||||
},
|
||||
"coverage": "GA4 reporting rows; inspect metadata for thresholding",
|
||||
"metadata": {
|
||||
"timeZone": "America/Los_Angeles",
|
||||
"currencyCode": "USD"
|
||||
},
|
||||
"periodEnd": "2026-10-25",
|
||||
"httpStatus": 200,
|
||||
"observedAt": "2026-10-28T07:18:43.791790+00:00",
|
||||
"periodStart": "2026-10-12"
|
||||
},
|
||||
"gsc": {
|
||||
"rows": [
|
||||
{
|
||||
"ctr": 0.15625,
|
||||
"keys": [
|
||||
"https://canina.ir/blog/dog-black-seed-nigella-sativa"
|
||||
],
|
||||
"clicks": 5,
|
||||
"position": 3.4375,
|
||||
"impressions": 32
|
||||
}
|
||||
],
|
||||
"source": "GSC",
|
||||
"status": "available",
|
||||
"metrics": {
|
||||
"clicks": 5,
|
||||
"position": 3.4375,
|
||||
"impressions": 32
|
||||
},
|
||||
"coverage": "GSC top rows; anonymized queries omitted",
|
||||
"metadata": {},
|
||||
"periodEnd": "2026-10-25",
|
||||
"httpStatus": 200,
|
||||
"observedAt": "2026-10-28T07:18:42.716059+00:00",
|
||||
"periodStart": "2026-10-12"
|
||||
},
|
||||
"clarity": {
|
||||
"error": "reporting_credentials_not_configured",
|
||||
"status": "unavailable",
|
||||
"metrics": null,
|
||||
"observedAt": "2026-10-28T07:18:43.791896+00:00"
|
||||
},
|
||||
"commerce": {
|
||||
"error": "article_to_order_linkage_not_verified",
|
||||
"status": "unavailable",
|
||||
"metrics": null,
|
||||
"observedAt": "2026-10-28T07:18:43.791906+00:00"
|
||||
},
|
||||
"gscQueries": {
|
||||
"rows": [],
|
||||
"source": "GSC",
|
||||
"status": "no_data",
|
||||
"metrics": null,
|
||||
"coverage": "GSC top rows; anonymized queries omitted",
|
||||
"metadata": {},
|
||||
"periodEnd": "2026-10-25",
|
||||
"httpStatus": 200,
|
||||
"observedAt": "2026-10-28T07:18:42.979366+00:00",
|
||||
"periodStart": "2026-10-12"
|
||||
}
|
||||
},
|
||||
"feedbackVersion": 1,
|
||||
"periodAlignment": "same date labels; GSC Pacific and GA4 property timezone may differ; not session-level alignment",
|
||||
"mandatoryDirective": "Measure only; do not publish"
|
||||
},
|
||||
"commerce": {
|
||||
"error": "article_to_order_linkage_not_verified",
|
||||
"source": "commerce",
|
||||
"status": "unavailable",
|
||||
"fetchedAt": "2026-10-28T07:18:43.791906+00:00"
|
||||
}
|
||||
},
|
||||
"ingestionKey": "bb6b6afac66b9fc1664dd7c8e9143e80d1d2d04b4cb064aeed7068648f03e625",
|
||||
"gscImpressions": 32,
|
||||
"gscClicks": 5,
|
||||
"gscPosition": 3.4375,
|
||||
"ga4ActiveUsers": 2,
|
||||
"ga4AvgEngagementSec": 16.5,
|
||||
"ga4BounceRate": 0.5,
|
||||
"ga4ScrollDepthRate": null,
|
||||
"attributedOrdersCount": null,
|
||||
"attributedSalesValue": null,
|
||||
"actionRecommendation": null,
|
||||
"analysisNotes": null,
|
||||
"createdAt": "2026-10-28T07:18:43.874Z"
|
||||
},
|
||||
"productionWrites": false,
|
||||
"scheduled": false
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
# Worker scope and acceptance
|
||||
Bound to baseline specs/002-seo-performance-tracking; this local directory is not a Git worktree, so artifacts are local only, not committed/deployed. No backend/UI edits, cron or production writes authorized.
|
||||
Replace fabricated zeros with explicit unavailable/no_data and null. Collect all findings by pagination, exact canonical URL Google matching, same inclusive date period ending UTC today minus three days (GSC final data; GA4 property timezone retained separately). Google successful API response, not session construction, proves access. GSC top-row reporting is not exhaustive keyword coverage. No causal commerce attribution without source linkage. Store snapshots/outbox durably and deduplicate period+revision; feedback revision binds mandatory directive and backend must reject stale revisions. No invented optimization thresholds: insufficient evidence until approved policy and comparison data exist.
|
||||
Plan: injectable REST collectors; local SQLite snapshot/outbox; backend adapter with strict pagination; explicit opt-in delivery, local read-only probe mode. Failure errors sanitized to class/status only. Tests precede implementation (9 RED failures observed). Test doubles are confined to tests.
|
||||
Tasks: implement collectors/store/orchestrator; test authentication/429/timeouts/partial pages and reruns; live read-only Google probe; backend integration contract pending sibling; document blockers and acceptance evidence. No automatic deployment.
|
||||
@@ -0,0 +1,190 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Read-only collection by default. Local durable snapshots; delivery is explicit."""
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
from urllib.parse import quote, urlsplit
|
||||
import requests
|
||||
|
||||
GSC_CREDS_PATH='/opt/data/credentials/gsc-service-account.json'
|
||||
GA4_PROPERTY_ID='552453291'
|
||||
GSC_SITE='sc-domain:canina.ir'
|
||||
|
||||
def now(): return dt.datetime.now(dt.timezone.utc).isoformat()
|
||||
def period(today=None,days=14):
|
||||
if days<1: raise ValueError('days must be positive')
|
||||
end=(today or dt.datetime.now(dt.timezone.utc).date())-dt.timedelta(days=3)
|
||||
return str(end-dt.timedelta(days=days-1)),str(end)
|
||||
def digest(value): return hashlib.sha256(json.dumps(value,sort_keys=True,ensure_ascii=False).encode()).hexdigest()
|
||||
def get_google_session():
|
||||
from google.oauth2 import service_account
|
||||
from google.auth.transport.requests import AuthorizedSession
|
||||
return AuthorizedSession(service_account.Credentials.from_service_account_file(GSC_CREDS_PATH,scopes=['https://www.googleapis.com/auth/webmasters.readonly','https://www.googleapis.com/auth/analytics.readonly']))
|
||||
def unavailable(reason): return {'status':'unavailable','metrics':None,'error':reason,'observedAt':now()}
|
||||
|
||||
class GoogleCollector:
|
||||
def __init__(self,session,page_size=25000): self.session,self.page_size=session,page_size
|
||||
def collect(self,source,url,body,dates):
|
||||
rows=[]; offset=0; metadata={}; seen=set()
|
||||
try:
|
||||
while True:
|
||||
payload=dict(body)
|
||||
if source=='GSC': payload.update(rowLimit=self.page_size,startRow=offset)
|
||||
else: payload.update(limit=str(self.page_size),offset=str(offset))
|
||||
response=self.session.post(url,json=payload,timeout=30)
|
||||
if response.status_code!=200: return unavailable(f'http_{response.status_code}')
|
||||
data=response.json(); batch=data.get('rows',[])
|
||||
if not isinstance(batch,list): raise ValueError('invalid rows')
|
||||
metadata.update(data.get('metadata',{}))
|
||||
if batch:
|
||||
signature=digest(batch)
|
||||
if signature in seen: raise ValueError('repeated pagination')
|
||||
seen.add(signature)
|
||||
rows.extend(batch); offset+=len(batch)
|
||||
if source=='GA4':
|
||||
if offset>=int(data.get('rowCount',offset)): break
|
||||
if not batch: raise ValueError('incomplete pagination')
|
||||
elif len(batch)<self.page_size: break
|
||||
return {'status':'available' if rows else 'no_data','metrics':None,'rows':rows,'periodStart':dates[0],'periodEnd':dates[1],'observedAt':now(),'metadata':metadata,'coverage':'GSC top rows; anonymized queries omitted' if source=='GSC' else 'GA4 reporting rows; inspect metadata for thresholding','source':source,'httpStatus':200}
|
||||
except Exception as exc:
|
||||
return unavailable(type(exc).__name__)
|
||||
def gsc(self,url,dates):
|
||||
result = self.collect('GSC',f'https://www.googleapis.com/webmasters/v3/sites/{quote(GSC_SITE,safe="")}/searchAnalytics/query',{'startDate':dates[0],'endDate':dates[1],'dataState':'final','dimensions':['query'],'dimensionFilterGroups':[{'filters':[{'dimension':'page','operator':'equals','expression':url}]}]},dates)
|
||||
return result
|
||||
def gsc_page(self,url,dates):
|
||||
result=self.collect('GSC',f'https://www.googleapis.com/webmasters/v3/sites/{quote(GSC_SITE,safe="")}/searchAnalytics/query',{'startDate':dates[0],'endDate':dates[1],'dataState':'final','dimensions':['page'],'dimensionFilterGroups':[{'filters':[{'dimension':'page','operator':'equals','expression':url}]}]},dates)
|
||||
if result['status']=='available':
|
||||
try:
|
||||
row=result['rows'][0]
|
||||
if len(result['rows'])!=1 or row['keys']!=[url]: raise ValueError('exact URL mismatch')
|
||||
result['metrics']={k:row[k] for k in ('clicks','impressions','position')}
|
||||
except Exception as exc: return unavailable(type(exc).__name__)
|
||||
return result
|
||||
def ga4(self,url,dates):
|
||||
result=self.collect('GA4',f'https://analyticsdata.googleapis.com/v1beta/properties/{GA4_PROPERTY_ID}:runReport',{'dateRanges':[{'startDate':dates[0],'endDate':dates[1]}],'dimensions':[{'name':'pageLocation'}],'metrics':[{'name':'activeUsers'},{'name':'userEngagementDuration'},{'name':'bounceRate'}],'dimensionFilter':{'filter':{'fieldName':'pageLocation','stringFilter':{'matchType':'EXACT','value':url,'caseSensitive':True}}}},dates)
|
||||
if result['status']=='available':
|
||||
try:
|
||||
rows=result['rows']
|
||||
if len(rows)!=1 or rows[0]['dimensionValues'][0]['value']!=url: raise ValueError('exact URL mismatch')
|
||||
vals=rows[0]['metricValues']; users=int(vals[0]['value']); duration=float(vals[1]['value'])
|
||||
result['metrics']={'activeUsers':users,'userEngagementDuration':duration,'avgEngagementSec':duration/users if users else None,'bounceRate':float(vals[2]['value']),'scrollDepthRate':None}
|
||||
except Exception as exc: return unavailable(type(exc).__name__)
|
||||
return result
|
||||
|
||||
class Store:
|
||||
def __init__(self,path):
|
||||
self.db=sqlite3.connect(path)
|
||||
self.db.execute('CREATE TABLE IF NOT EXISTS snapshots (key TEXT PRIMARY KEY, payload TEXT NOT NULL, delivered INTEGER DEFAULT 0)')
|
||||
self.db.execute('CREATE TABLE IF NOT EXISTS outbox (key TEXT PRIMARY KEY,payload TEXT NOT NULL, delivered INTEGER DEFAULT 0)')
|
||||
def snapshot(self,payload):
|
||||
key=digest([payload.get(k) for k in ('findingId','periodStart','periodEnd','feedbackRevision')]); payload=dict(payload,idempotencyKey=key)
|
||||
with self.db:
|
||||
cur=self.db.execute('INSERT OR IGNORE INTO snapshots(key,payload) VALUES (?,?)',(key,json.dumps(payload,ensure_ascii=False)))
|
||||
return bool(cur.rowcount)
|
||||
def alert(self,source,error,finding):
|
||||
payload={'source':source,'error':error,'findingId':finding,'department':'TECHNICAL_SEO' if source=='GSC' else 'ENGINEERING','observedAt':now(),'status':'pending'}
|
||||
key=digest([source,error,finding,dt.datetime.now(dt.timezone.utc).date().isoformat()])
|
||||
with self.db: self.db.execute('INSERT OR IGNORE INTO outbox(key,payload) VALUES (?,?)',(key,json.dumps(payload)))
|
||||
def pending(self): return self.db.execute('SELECT key,payload FROM outbox WHERE delivered=0').fetchall()
|
||||
|
||||
class Backend:
|
||||
def __init__(self,base,session): self.base,self.session=base.rstrip('/')+'/api/admin/seo-tracking',session
|
||||
def findings(self):
|
||||
page=1; found={}; total=None
|
||||
while True:
|
||||
r=self.session.get(self.base,params={'page':page,'limit':100},timeout=30)
|
||||
if r.status_code!=200: raise RuntimeError(f'backend_http_{r.status_code}')
|
||||
data=r.json(); meta=data['meta']; batch=data['data']
|
||||
if int(meta['page'])!=page: raise ValueError('pagination mismatch')
|
||||
if total is None: total=int(meta['total'])
|
||||
if int(meta['total'])!=total: raise ValueError('findings changed during pagination')
|
||||
for item in batch:
|
||||
if item['id'] in found: raise ValueError('duplicate finding')
|
||||
found[item['id']]=item
|
||||
if page>=int(meta['totalPages']): break
|
||||
if not batch: raise ValueError('incomplete pagination')
|
||||
page+=1
|
||||
if len(found)!=total: raise ValueError('finding count mismatch')
|
||||
return list(found.values())
|
||||
|
||||
def decision(finding,sources):
|
||||
return {'action':'INSUFFICIENT_EVIDENCE','reason':'No approved comparative decision policy; unavailable signals cannot justify scaling or pruning. Correlation is not keyword-caused sales.','mandatoryDirective':finding.get('ceoFeedback'),'feedbackRevision':finding.get('feedbackVersion',finding.get('feedbackRevision',finding.get('feedbackSubmittedAt')))}
|
||||
|
||||
def collect_finding(finding,collector,dates,store):
|
||||
blog=finding.get('blog') or {}; url=finding.get('articleUrl') or blog.get('canonicalUrl')
|
||||
if not url and blog.get('slug'): url='https://canina.ir/blog/'+quote(blog['slug'],safe='')
|
||||
if not url or urlsplit(url).hostname!='canina.ir' or urlsplit(url).scheme!='https':
|
||||
store.alert('BACKEND','missing_or_invalid_article_url',finding['id']); return None
|
||||
sources={'gsc':collector.gsc_page(url,dates),'gscQueries':collector.gsc(url,dates),'ga4':collector.ga4(url,dates),'clarity':unavailable('reporting_credentials_not_configured'),'commerce':unavailable('article_to_order_linkage_not_verified')}
|
||||
payload={'findingId':finding['id'],'articleUrl':url,'periodStart':dates[0],'periodEnd':dates[1],'feedbackRevision':finding.get('feedbackVersion',finding.get('feedbackRevision',finding.get('feedbackSubmittedAt'))),'sources':sources,'decision':decision(finding,sources),'observedAt':now(),'periodAlignment':'same date labels; GSC Pacific and GA4 property timezone may differ; not session-level alignment'}
|
||||
for source,data in sources.items():
|
||||
if data['status']=='unavailable': store.alert(source.upper(),data['error'],finding['id'])
|
||||
store.snapshot(payload); return payload
|
||||
|
||||
def snapshot_body(payload,key):
|
||||
sources=payload['sources']; metadata={}; body={'ingestionKey':key,'periodStart':payload['periodStart']+'T00:00:00.000Z','periodEnd':str(dt.date.fromisoformat(payload['periodEnd'])+dt.timedelta(days=1))+'T00:00:00.000Z'}
|
||||
mappings={'gsc':{'clicks':'gscClicks','impressions':'gscImpressions','position':'gscPosition'},'ga4':{'activeUsers':'ga4ActiveUsers','avgEngagementSec':'ga4AvgEngagementSec','bounceRate':'ga4BounceRate'},'commerce':{'orders':'attributedOrdersCount','revenue':'attributedSalesValue'}}
|
||||
for source,mapping in mappings.items():
|
||||
data=sources.get(source) or unavailable('missing_source'); metrics=data.get('metrics')
|
||||
available=data['status']=='available' and metrics is not None and all(metrics.get(k) is not None for k in mapping)
|
||||
metadata[source]={'status':'available' if available else 'unavailable','source':source,'fetchedAt':data.get('observedAt',now())}
|
||||
if available:
|
||||
for k,v in mapping.items(): body[v]=metrics[k]
|
||||
else: metadata[source]['error']=data.get('error','no_reported_metrics')
|
||||
metadata['worker']={'feedbackVersion':payload.get('feedbackRevision'),'mandatoryDirective':payload.get('decision',{}).get('mandatoryDirective'),'periodAlignment':payload.get('periodAlignment'),'sources':sources}
|
||||
body['sourceMetadata']=metadata
|
||||
return body
|
||||
|
||||
def deliver_snapshots(store,backend):
|
||||
"""Explicit mutation function, never invoked in read-only CLI. Backend dedupes ingestionKey."""
|
||||
count=0
|
||||
for key,raw in store.db.execute('SELECT key,payload FROM snapshots WHERE delivered=0').fetchall():
|
||||
payload=json.loads(raw)
|
||||
try:
|
||||
response=backend.session.get(backend.base+'/'+payload['findingId'],timeout=30)
|
||||
if response.status_code!=200: raise RuntimeError('detail_unavailable')
|
||||
if response.json().get('feedbackVersion')!=payload['feedbackRevision']:
|
||||
store.alert('BACKEND','stale_feedback',payload['findingId']); continue
|
||||
# Measurement snapshots do not apply directives or advance article status.
|
||||
response=backend.session.post(backend.base+'/'+payload['findingId']+'/snapshots',json=snapshot_body(payload,key),timeout=30)
|
||||
if response.status_code not in (200,201): raise RuntimeError(f'http_{response.status_code}')
|
||||
with store.db: store.db.execute('UPDATE snapshots SET delivered=1 WHERE key=?',(key,))
|
||||
count+=1
|
||||
except Exception as exc: store.alert('BACKEND',type(exc).__name__,payload['findingId'])
|
||||
return count
|
||||
|
||||
def deliver_alerts(store,session,endpoint):
|
||||
"""Send structured department events to configured receiver; require durable receiver ack."""
|
||||
count=0
|
||||
for key,raw in store.pending():
|
||||
try:
|
||||
response=session.post(endpoint,json=dict(json.loads(raw),idempotencyKey=key),timeout=30)
|
||||
if response.status_code not in (200,201,202): continue
|
||||
with store.db: store.db.execute('UPDATE outbox SET delivered=1 WHERE key=?',(key,))
|
||||
count+=1
|
||||
except Exception: pass # event remains pending; no secret-bearing exception logging
|
||||
return count
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser(); p.add_argument('--backend'); p.add_argument('--probe-url'); p.add_argument('--days',type=int,default=14); p.add_argument('--state',default='tracker.sqlite'); p.add_argument('--receipt',default='receipt.json'); args=p.parse_args()
|
||||
store=Store(args.state); dates=period(days=args.days); results=[]
|
||||
try: collector=GoogleCollector(get_google_session())
|
||||
except Exception as exc:
|
||||
store.alert('GOOGLE',type(exc).__name__,'global'); Path(args.receipt).write_text(json.dumps({'status':'blocked','error':type(exc).__name__,'observedAt':now()})); return 1
|
||||
if args.probe_url: findings=[{'id':'read-only-probe','articleUrl':args.probe_url}]
|
||||
elif args.backend:
|
||||
session=requests.Session(); token=os.getenv('CANINA_ADMIN_TOKEN')
|
||||
if token: session.headers['Authorization']='Bearer '+token
|
||||
try: findings=Backend(args.backend,session).findings()
|
||||
except Exception as exc:
|
||||
store.alert('BACKEND',type(exc).__name__,'global'); Path(args.receipt).write_text(json.dumps({'status':'blocked','error':type(exc).__name__,'observedAt':now()})); return 1
|
||||
else: p.error('--backend or --probe-url required')
|
||||
for finding in findings:
|
||||
if finding.get('blog') or finding.get('articleUrl'): results.append(collect_finding(finding,collector,dates,store))
|
||||
Path(args.receipt).write_text(json.dumps({'observedAt':now(),'period':dates,'findingCount':len(findings),'collectedCount':sum(x is not None for x in results),'results':results,'remoteWrites':False},ensure_ascii=False,indent=2))
|
||||
return 0
|
||||
if __name__=='__main__': raise SystemExit(main())
|
||||
@@ -0,0 +1,21 @@
|
||||
import unittest, tempfile, json
|
||||
from pathlib import Path
|
||||
from unittest.mock import Mock
|
||||
import seo_performance_tracker as w
|
||||
|
||||
class DeliveryTests(unittest.TestCase):
|
||||
def test_retry_and_ack(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
st=w.Store(Path(d)/'s.db');st.alert('GSC','http_403','f'); session=Mock(); session.post.return_value.status_code=429
|
||||
self.assertEqual(w.deliver_alerts(st,session,'http://localhost/alerts'),0);self.assertEqual(len(st.pending()),1)
|
||||
session.post.return_value.status_code=202
|
||||
self.assertEqual(w.deliver_alerts(st,session,'http://localhost/alerts'),1);self.assertEqual(w.deliver_alerts(st,session,'http://localhost/alerts'),0)
|
||||
def test_contract_null(self):
|
||||
p={'findingId':'f','periodStart':'2026-10-01','periodEnd':'2026-10-14','feedbackRevision':2,'sources':{'gsc':w.unavailable('http_403'),'ga4':w.unavailable('Timeout'),'commerce':w.unavailable('missing')}}
|
||||
x=w.snapshot_body(p,'key'); self.assertEqual(x['ingestionKey'],'key');self.assertNotIn('gscClicks',x);self.assertEqual(x['sourceMetadata']['gsc']['status'],'unavailable');self.assertNotIn('actionRecommendation',x)
|
||||
def test_stale_feedback_blocks_delivery(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
st=w.Store(Path(d)/'s.db');st.snapshot({'findingId':'f','periodStart':'2026-10-01','periodEnd':'2026-10-14','feedbackRevision':2,'sources':{}})
|
||||
s=Mock();s.get.return_value.status_code=200;s.get.return_value.json.return_value={'feedbackVersion':3}
|
||||
self.assertEqual(w.deliver_snapshots(st,w.Backend('http://localhost',s)),0);s.post.assert_not_called();self.assertEqual(len(st.pending()),1)
|
||||
if __name__=='__main__':unittest.main()
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Local HTTP integration; synthetic fixture responses exist only in this test."""
|
||||
import unittest,threading,json,tempfile
|
||||
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
import requests
|
||||
import seo_performance_tracker as w
|
||||
class Integration(unittest.TestCase):
|
||||
def test_http_pagination_collection_and_delivery(self):
|
||||
calls=[]
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def log_message(self,*a): pass
|
||||
def do_GET(self):
|
||||
if '?' in self.path:
|
||||
p=1 if 'page=1&' in self.path else 2
|
||||
data={'data':[{'id':str(p)}],'meta':{'page':p,'total':2,'totalPages':2}}
|
||||
else:data={'feedbackVersion':1}
|
||||
self.send_response(200);self.end_headers();self.wfile.write(json.dumps(data).encode())
|
||||
def do_POST(self):
|
||||
calls.append(json.loads(self.rfile.read(int(self.headers['Content-Length']))));self.send_response(201);self.end_headers();self.wfile.write(b'{}')
|
||||
server=ThreadingHTTPServer(('127.0.0.1',0),Handler);thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()
|
||||
try:
|
||||
backend=w.Backend('http://127.0.0.1:'+str(server.server_port),requests.Session());self.assertEqual(len(backend.findings()),2)
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
store=w.Store(Path(d)/'s');store.snapshot({'findingId':'1','periodStart':'2026-10-01','periodEnd':'2026-10-14','feedbackRevision':1,'sources':{}})
|
||||
self.assertEqual(w.deliver_snapshots(store,backend),1);self.assertEqual(w.deliver_snapshots(store,backend),0)
|
||||
self.assertEqual(calls[0]['sourceMetadata']['commerce']['status'],'unavailable');store.db.close()
|
||||
finally:server.shutdown();server.server_close();thread.join()
|
||||
@@ -0,0 +1,9 @@
|
||||
"""Regression: arbitrary legacy thresholds must not drive editorial changes."""
|
||||
import unittest
|
||||
from seo_performance_tracker import decision
|
||||
class TestConservativeDecisions(unittest.TestCase):
|
||||
def test_aggregate_sales_do_not_prove_keyword_causality(self):
|
||||
self.assertEqual(decision({}, {'orders': 99, 'clicks': 1000})['action'],'INSUFFICIENT_EVIDENCE')
|
||||
def test_directive_is_carried_without_inventing_action(self):
|
||||
r=decision({'ceoFeedback':'Revise citations','feedbackVersion':4},{})
|
||||
self.assertEqual(r['mandatoryDirective'],'Revise citations');self.assertEqual(r['feedbackRevision'],4)
|
||||
@@ -0,0 +1,55 @@
|
||||
import unittest, tempfile, datetime, json
|
||||
from pathlib import Path
|
||||
from unittest.mock import Mock
|
||||
import requests
|
||||
import seo_performance_tracker as w
|
||||
|
||||
class WorkerTests(unittest.TestCase):
|
||||
def test_period(self):
|
||||
self.assertEqual(w.period(datetime.date(2026,10,28),14),('2026-10-12','2026-10-25'))
|
||||
def test_unavailable_is_null(self):
|
||||
for status in (401,403,429,500):
|
||||
s=Mock(); s.post.return_value.status_code=status
|
||||
result=w.GoogleCollector(s).gsc('https://canina.ir/blog/a',('2026-10-01','2026-10-14'))
|
||||
self.assertEqual(result['status'],'unavailable'); self.assertIsNone(result['metrics']); self.assertEqual(result['error'],f'http_{status}')
|
||||
def test_timeout(self):
|
||||
s=Mock(); s.post.side_effect=requests.Timeout('secret response')
|
||||
self.assertEqual(w.GoogleCollector(s).ga4('https://canina.ir/blog/a',('2026-10-01','2026-10-14'))['error'],'Timeout')
|
||||
def test_exact_ga4(self):
|
||||
s=Mock(); s.post.return_value.status_code=200; s.post.return_value.json.return_value={'rows':[],'rowCount':0}
|
||||
r=w.GoogleCollector(s).ga4('https://canina.ir/blog/a',('2026-10-01','2026-10-14'))
|
||||
f=s.post.call_args.kwargs['json']['dimensionFilter']['filter']
|
||||
self.assertEqual(f['fieldName'],'pageLocation'); self.assertEqual(f['stringFilter']['matchType'],'EXACT'); self.assertEqual(r['status'],'no_data'); self.assertIsNone(r['metrics'])
|
||||
def test_gsc_pagination(self):
|
||||
s=Mock(); a=Mock(status_code=200); b=Mock(status_code=200)
|
||||
a.json.return_value={'rows':[{'keys':['x'],'clicks':1,'impressions':2,'position':4,'ctr':.5}]}; b.json.return_value={}
|
||||
s.post.side_effect=[a,b]
|
||||
r=w.GoogleCollector(s,page_size=1).gsc('https://canina.ir/blog/a',('2026-10-01','2026-10-14'))
|
||||
self.assertEqual(s.post.call_count,2); self.assertEqual(r['rows'][0]['clicks'],1)
|
||||
def test_partial_pagination_discards_aggregates(self):
|
||||
s=Mock(); a=Mock(status_code=200); a.json.return_value={'rows':[{'keys':['x']}]}; s.post.side_effect=[a,requests.Timeout()]
|
||||
r=w.GoogleCollector(s,page_size=1).gsc('https://canina.ir/blog/a',('2026-10-01','2026-10-14'))
|
||||
self.assertEqual(r['status'],'unavailable'); self.assertIsNone(r['metrics'])
|
||||
def test_idempotent_local_and_revision(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
store=w.Store(Path(d)/'state.sqlite'); p={'findingId':'1','periodStart':'a','periodEnd':'b','feedbackRevision':1}
|
||||
self.assertTrue(store.snapshot(p)); self.assertFalse(store.snapshot(p)); p['feedbackRevision']=2; self.assertTrue(store.snapshot(p))
|
||||
store.alert('GSC','http_403','1');store.alert('GSC','http_403','1');self.assertEqual(len(store.pending()),1)
|
||||
def test_conservative_and_directives(self):
|
||||
r=w.decision({'ceoFeedback':'Do not scale','feedbackRevision':3},{})
|
||||
self.assertEqual(r['action'],'INSUFFICIENT_EVIDENCE'); self.assertEqual(r['mandatoryDirective'],'Do not scale'); self.assertEqual(r['feedbackRevision'],3)
|
||||
def test_ga4_pagination_and_row_count(self):
|
||||
s=Mock(); a=Mock(status_code=200); b=Mock(status_code=200)
|
||||
a.json.return_value={'rows':[{'dimensionValues':[{'value':'x'}]}],'rowCount':2};b.json.return_value={'rows':[{'dimensionValues':[{'value':'y'}]}],'rowCount':2};s.post.side_effect=[a,b]
|
||||
result=w.GoogleCollector(s,page_size=1).collect('GA4','https://example.invalid',{},('a','b'))
|
||||
self.assertEqual(len(result['rows']),2);self.assertEqual(s.post.call_args.kwargs['json']['offset'],'1')
|
||||
def test_malformed_json_sanitized(self):
|
||||
s=Mock();s.post.return_value.status_code=200;s.post.return_value.json.side_effect=ValueError('secret')
|
||||
r=w.GoogleCollector(s).gsc('https://canina.ir/blog/a',('a','b'))
|
||||
self.assertEqual(r['error'],'ValueError');self.assertNotIn('secret',json.dumps(r))
|
||||
def test_backend_pagination(self):
|
||||
s=Mock(); a=Mock(status_code=200);b=Mock(status_code=200)
|
||||
a.json.return_value={'data':[{'id':'1'}],'meta':{'page':1,'totalPages':2,'total':2}}; b.json.return_value={'data':[{'id':'2'}],'meta':{'page':2,'totalPages':2,'total':2}}
|
||||
s.get.side_effect=[a,b]; self.assertEqual(len(w.Backend('http://localhost',s).findings()),2)
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
@@ -0,0 +1,82 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Opt-in acceptance: called by Jest against disposable local Nest/PG only."""
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import requests
|
||||
import seo_performance_tracker as w
|
||||
|
||||
|
||||
def main():
|
||||
base = os.environ['SEO_LOCAL_BASE']
|
||||
assert base.startswith('http://127.0.0.1:'), 'local test server only'
|
||||
session = requests.Session()
|
||||
session.headers['Authorization'] = 'Bearer ' + os.environ['CANINA_ADMIN_TOKEN']
|
||||
backend = w.Backend(base, session)
|
||||
checks = {}
|
||||
|
||||
def call(method, suffix='', body=None, expected=200):
|
||||
response = session.request(method, backend.base + suffix, json=body, timeout=30)
|
||||
assert response.status_code == expected, (method, suffix, response.status_code, response.text)
|
||||
return response.json()
|
||||
|
||||
assert requests.get(backend.base, timeout=30).status_code == 401
|
||||
assert requests.get(backend.base, headers={'Authorization': 'Bearer invalid'}, timeout=30).status_code == 401
|
||||
assert requests.get(backend.base, headers={'Authorization': 'Bearer ' + os.environ['SEO_CUSTOMER_TOKEN']}, timeout=30).status_code == 403
|
||||
checks['missing_invalid_nonadmin_denied'] = True
|
||||
created = call('POST', body={
|
||||
'title': 'Disposable blackseed research acceptance', 'hypothesis': 'Integration fixture only',
|
||||
'primaryKeyword': 'blackseed', 'targetKeywords': ['blackseed'],
|
||||
'expectedOutcome': 'Measure existing published article without publishing',
|
||||
'blogId': os.environ['SEO_FIXTURE_BLOG_ID'],
|
||||
}, expected=201)
|
||||
suffix = '/' + created['id']
|
||||
feedback = call('PATCH', suffix + '/feedback', {'expectedVersion': 0, 'feedback': 'Measure only; do not publish'})
|
||||
assert (feedback['version'], feedback['feedbackVersion']) == (1, 1)
|
||||
call('PATCH', suffix + '/action', {'expectedVersion': 1, 'feedbackVersion': 0, 'actionId': 'disposable-fixture'}, expected=409)
|
||||
action = call('PATCH', suffix + '/action', {'expectedVersion': 1, 'feedbackVersion': 1, 'actionId': 'disposable-fixture', 'status': 'ACTIONED'})
|
||||
assert (action['version'], action['actionId']) == (2, 'disposable-fixture')
|
||||
checks['feedback_stale_action_and_correct_link'] = True
|
||||
finding = next(f for f in backend.findings() if f['id'] == created['id'])
|
||||
assert finding['blog']['slug'] == 'dog-black-seed-nigella-sativa'
|
||||
with tempfile.TemporaryDirectory(prefix='seo-worker-') as directory:
|
||||
store = w.Store(Path(directory) / 'state.sqlite')
|
||||
collector = w.GoogleCollector(w.get_google_session())
|
||||
payload = w.collect_finding(finding, collector, w.period(), store)
|
||||
assert payload is not None
|
||||
# Reopen durable worker state before delivery to exercise process restart semantics.
|
||||
store.db.close()
|
||||
store = w.Store(Path(directory) / 'state.sqlite')
|
||||
assert w.deliver_snapshots(store, backend) == 1
|
||||
assert w.deliver_snapshots(store, backend) == 0
|
||||
history = call('GET', suffix)
|
||||
assert len(history['snapshots']) == 1
|
||||
first = history['snapshots'][0]
|
||||
assert first['attributedSalesValue'] is None
|
||||
assert first['ga4ScrollDepthRate'] is None
|
||||
key, raw = store.db.execute('SELECT key,payload FROM snapshots').fetchone()
|
||||
body = w.snapshot_body(json.loads(raw), key)
|
||||
# Actual network retry of identical immutable request, not just local delivered flag.
|
||||
retried = call('POST', suffix + '/snapshots', body, expected=201)
|
||||
assert retried['id'] == first['id']
|
||||
call('POST', suffix + '/snapshots', dict(body, analysisNotes='changed payload'), expected=409)
|
||||
assert len(call('GET', suffix)['snapshots']) == 1
|
||||
assert not store.snapshot(payload)
|
||||
checks['worker_snapshot_history_network_retry_conflict'] = True
|
||||
receipt = {
|
||||
'observedAt': w.now(), 'backend': 'disposable local Nest module + real PostgreSQL',
|
||||
'auth': 'test-only JWT strategy with real JwtAuthGuard/RolesGuard; NOT production identity/API-key',
|
||||
'checks': checks, 'google': payload, 'persistedSnapshot': first,
|
||||
'productionWrites': False, 'scheduled': False,
|
||||
}
|
||||
destination = os.environ.get('SEO_HTTP_RECEIPT')
|
||||
if destination:
|
||||
Path(destination).parent.mkdir(parents=True, exist_ok=True)
|
||||
Path(destination).write_text(json.dumps(receipt, ensure_ascii=False, indent=2))
|
||||
store.db.close()
|
||||
print('REAL_HTTP_ACCEPTANCE_PASSED ' + json.dumps(checks))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in new issue
Block a user