From 5750e1589ea3c66a9189f3ed3979f953d368dd97 Mon Sep 17 00:00:00 2001 From: Canina Automation Date: Wed, 28 Oct 2026 07:18:40 +0000 Subject: [PATCH] feat(content): add dormant versioned research directive publisher consumer --- scripts/content-research/README.md | 26 +++ scripts/content-research/consumer.py | 149 +++++++++++++++++ scripts/content-research/daily_publisher.py | 16 ++ scripts/content-research/test_consumer.py | 171 ++++++++++++++++++++ specs/content-research-consumer/plan.md | 6 + specs/content-research-consumer/spec.md | 12 ++ specs/content-research-consumer/tasks.md | 5 + 7 files changed, 385 insertions(+) create mode 100644 scripts/content-research/README.md create mode 100644 scripts/content-research/consumer.py create mode 100644 scripts/content-research/daily_publisher.py create mode 100644 scripts/content-research/test_consumer.py create mode 100644 specs/content-research-consumer/plan.md create mode 100644 specs/content-research-consumer/spec.md create mode 100644 specs/content-research-consumer/tasks.md diff --git a/scripts/content-research/README.md b/scripts/content-research/README.md new file mode 100644 index 00000000..267b996b --- /dev/null +++ b/scripts/content-research/README.md @@ -0,0 +1,26 @@ +# Dormant daily publisher research/directive consumer + +No cron or active runtime is changed. The actual publisher is agent-driven by `/opt/data/publishers/e4c9b52eb040/cron-prompt.txt` (research line 11, draft/reviews line 13, guarded Admin creation line 17). Its directory and parent batch engine are outside Git. This repository previously contained no matching daily-publisher module. `daily_publisher.run_new` is the versioned integration boundary, not a claim of live enforcement. + +## Authorized future caller wiring + +Provision these modules through a reviewed Git/Stage/runtime process, then have the existing owner call `run_new(Consumer(journal, AdminTransport(verified_admin_origin, approved_bearer)), stable_run_id, finding_payload, Content_callback, independent_review_callback, existing_guarded_NEW_create_callback)`. + +- finding_payload uses the backend CreateFindingDto contract (title, hypothesis, targetKeywords, primaryKeyword, expectedOutcome, optional metric goal). No article is created by the adapter itself. +- Content callback receives the entire saved/fetched finding, including exact CEO free-text feedback/version. Save immutable draft artifacts; preserve the existing medical/SEO/QA chain. +- Independent reviewer callback receives finding, draft and both SHA256 digests. A nonempty directive requires reviewer identity, evidence artifact URI/path, exact directiveHash and draftHash. Review must quote the directive and map every request to draft passages or explicit unresolved findings. The adapter verifies binding, not arbitrary semantic obedience or reviewer authenticity; trusted review orchestration remains essential. Empty feedback needs no CEO approval. No Gate1/2 is introduced. +- Existing create callback must retain publisher_state ownership, fresh inventory/relations/quota checks, nonempty backup, independent exact-hash QA and write-ahead intent. Return confirmed blog UUID only. Never wrap an unguarded HTTP create callback in production. +- Persist journal outside temporary storage, shared by all invocations, with stable run ID. It records consumed finding and feedback versions, review, draft digest and created identity. A concurrent invocation cannot enter create after a claim transition. Network ambiguity blocks retries. Existing daily quota guard still protects different run IDs and different journal files. +- `reconcile_link(run_id)` only links the saved confirmed blog, never creates another article. If feedback changed after create, retain the blog and route independent repair. No automatic resave/update. + +## Atomicity limit (not concealed) + +Backend PATCH action atomically checks `expectedVersion` AND `feedbackVersion`; conflict prevents the callback. It does **not** participate in the later Blog create transaction. CEO edits after claim and before actual creation can therefore race. The adapter detects changed feedback on post-create link, retains the article identity and blocks linking/recreation, but cannot roll back the article. Full no-stale-publication requires an additional backend transaction/lease contract on the actual Blog creation boundary. Do not claim that consumption acknowledgement guarantees free-text implementation or that this dormant adapter protects current cron publication. + +## Executed evidence + +- RED before implementation: unittest import failed because consumer module did not exist. +- `python -B -m unittest discover -s scripts/content-research -v`: **11 passed**, covering real integration entrypoint with isolated transport/callback fixtures, feedback before/after claim, optional feedback, exact receipt hashes, contention, idempotency, research timeout and publication ambiguity, link-only repair. +- Existing Git editorial suites: **2 passed, 3 skipped** (root-only ownership fixtures unavailable). +- Existing untouched runtime `python -B -m unittest discover -s /opt/data/publishers/e4c9b52eb040 -p test_publisher_state.py -v`: **3 passed, 2 errors**. Both existing tests call intent without the now-required independent QA fixture and fail `QA_REQUIRED: current-day independent exact-hash approval`. Runtime/test files were not edited; this is a pre-existing fixture drift, not a green regression claim. +- No live API request, publication, deployment, push, cron edit or activation. No backend/UI change. Tests establish local callback/API contract only, not live database/network acceptance. diff --git a/scripts/content-research/consumer.py b/scripts/content-research/consumer.py new file mode 100644 index 00000000..c56bd9e9 --- /dev/null +++ b/scripts/content-research/consumer.py @@ -0,0 +1,149 @@ +"""Dormant NEW-only publisher boundary. Never invokes a publisher on its own.""" +import hashlib +import json +import sqlite3 +import urllib.request +import urllib.error + +BASE = '/api/admin/seo-tracking' + + +class Blocked(RuntimeError): + pass + + +def digest(value): + return hashlib.sha256(json.dumps(value, sort_keys=True, ensure_ascii=False, + separators=(',', ':')).encode()).hexdigest() + + +class AdminTransport: + """Use approved short-lived bearer supplied by caller; no retry or secret logging.""" + def __init__(self, origin, bearer): + if not origin.startswith('https://'): + raise ValueError('HTTPS Admin origin required') + self.origin, self.bearer = origin.rstrip('/'), bearer + + def __call__(self, method, path, body=None): + if path != BASE and not path.startswith(BASE + '/'): + raise ValueError('Research endpoints only') + request = urllib.request.Request( + self.origin + path, method=method, + data=None if body is None else json.dumps(body).encode(), + headers={'Authorization': 'Bearer ' + self.bearer, + 'Content-Type': 'application/json'}) + # Never forward Authorization to redirects, even same-origin redirects. + class NoRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request(self, *args, **kwargs): + return None + try: + with urllib.request.build_opener(NoRedirect).open(request, timeout=30) as response: + return json.load(response) + except urllib.error.HTTPError as error: + raise Blocked('Admin HTTP ' + str(error.code)) from None + + +class Consumer: + def __init__(self, path, api): + self.path, self.api = path, api + with self.db() as db: + db.execute('CREATE TABLE IF NOT EXISTS runs (run TEXT PRIMARY KEY, state TEXT NOT NULL, finding TEXT, packet TEXT, draft_hash TEXT, review TEXT, blog TEXT)') + + def db(self): + db = sqlite3.connect(self.path, timeout=10) + db.row_factory = sqlite3.Row + return db + + def row(self, run): + with self.db() as db: + row = db.execute('SELECT * FROM runs WHERE run=?', (run,)).fetchone() + if row is None: + raise Blocked('Research must be saved before drafting') + return dict(row) + + def prepare(self, run, research): + with self.db() as db: + try: + db.execute('INSERT INTO runs(run,state) VALUES (?,?)', (run, 'research_intent')) + except sqlite3.IntegrityError: + row = self.row(run) + if row['state'] == 'prepared': + return json.loads(row['packet']) + raise Blocked('Existing research intent/outcome requires reconciliation') from None + # Persist intent before POST: timeout must never cause duplicate research. + finding = self.api('POST', BASE, research) + packet = self.api('GET', BASE + '/' + finding['id']) + self.check_packet(packet) + with self.db() as db: + db.execute('UPDATE runs SET state=?,finding=?,packet=? WHERE run=?', + ('prepared', packet['id'], json.dumps(packet), run)) + return packet + + @staticmethod + def check_packet(packet): + for key in ('version', 'feedbackVersion'): + if type(packet.get(key)) is not int or packet[key] < 0: + raise Blocked('Missing directive version') + if not isinstance(packet.get('id'), str) or not packet['id']: + raise Blocked('Missing finding identity') + if packet.get('ceoFeedback') is not None and not isinstance(packet['ceoFeedback'], str): + raise Blocked('Invalid directive') + + def publish(self, run, packet, draft, review, publisher_create): + """Caller must preserve existing ownership/QA/backup/quota/intent guards. + + packet is the complete research/directive supplied to Content before drafting. + review is an independent exact-hash receipt, NOT semantic-compliance proof. + """ + row = self.row(run) + if row['packet'] is None or digest(json.loads(row['packet'])) != digest(packet): + raise Blocked('Packet differs from saved directive') + if row['state'] == 'linked': + return row['blog'] + if row['state'] != 'prepared': + raise Blocked('In-flight/ambiguous action: reconcile, never recreate') + self.check_packet(packet) + if packet.get('ceoFeedback', None): + if not isinstance(review, dict) or not review.get('reviewer') or not review.get('evidence') or review.get('directiveHash') != digest(packet) or review.get('draftHash') != digest(draft): + raise Blocked('Exact directive/draft independent review evidence required') + with self.db() as db: + changed = db.execute('UPDATE runs SET state=?,draft_hash=?,review=? WHERE run=? AND state=?', + ('claiming', digest(draft), json.dumps(review), run, 'prepared')).rowcount + if changed != 1: + raise Blocked('Another consumer owns action') + # Atomic Backend predicate rejects edits between fetch and claim. + claim = self.api('PATCH', BASE + '/' + packet['id'] + '/action', + dict(expectedVersion=packet['version'], feedbackVersion=packet['feedbackVersion'], actionId=run)) + self.check_packet(claim) + if claim['feedbackVersion'] != packet['feedbackVersion']: + raise Blocked('Unexpected claim response') + with self.db() as db: + db.execute('UPDATE runs SET state=? WHERE run=?', ('publication_intent', run)) + blog = publisher_create(draft) # No retry on exception or uncertain result. + if not isinstance(blog, str) or not blog: + raise Blocked('Missing blog identity; reconcile publication intent') + with self.db() as db: + db.execute('UPDATE runs SET state=?,blog=? WHERE run=?', ('created', blog, run)) + self.reconcile_link(run) + return blog + + def reconcile_link(self, run): + row = self.row(run) + if row['state'] == 'linked': + return row['blog'] + if row['state'] != 'created' or not row['blog']: + raise Blocked('No confirmed created article; authenticated reconciliation required') + packet = json.loads(row['packet']) + current = self.api('GET', BASE + '/' + row['finding']) + self.check_packet(current) + if current['feedbackVersion'] != packet['feedbackVersion']: + raise Blocked('Directive changed after create: retain blog, independent repair required') + if current.get('blogId') not in (None, row['blog']): + raise Blocked('Finding already linked to another article') + if current.get('blogId') != row['blog']: + self.api('PATCH', BASE + '/' + row['finding'] + '/action', + dict(expectedVersion=current['version'], feedbackVersion=current['feedbackVersion'], + actionId=run, blogId=row['blog'])) + with self.db() as db: + db.execute('UPDATE runs SET state=? WHERE run=?', ('linked', run)) + return row['blog'] diff --git a/scripts/content-research/daily_publisher.py b/scripts/content-research/daily_publisher.py new file mode 100644 index 00000000..2e58fdcc --- /dev/null +++ b/scripts/content-research/daily_publisher.py @@ -0,0 +1,16 @@ +"""Versioned integration entrypoint; not installed in the active cron runtime. + +The supplied create_new callback MUST be the existing guarded NEW publisher, +including publisher_state ownership, quota, immutable QA, backup and intent. +No article update or automatic retry path is exposed here. +""" +from consumer import digest + + +def run_new(consumer, run_id, research, draft_content, independent_review, create_new): + packet = consumer.prepare(run_id, research) # Persist research BEFORE drafting. + draft = draft_content(packet) # Complete fetched CEO directive, not just a keyword. + # Reviewer receives exact immutable content/directive plus digests. A valid receipt + # establishes traceability, not a machine guarantee of free-text obedience. + receipt = independent_review(packet, draft, digest(packet), digest(draft)) + return consumer.publish(run_id, packet, draft, receipt, create_new) diff --git a/scripts/content-research/test_consumer.py b/scripts/content-research/test_consumer.py new file mode 100644 index 00000000..9a15c610 --- /dev/null +++ b/scripts/content-research/test_consumer.py @@ -0,0 +1,171 @@ +import tempfile +import unittest +from pathlib import Path +from consumer import Consumer, Blocked, digest + + +class API: + def __init__(self): + self.row = dict(id='finding', version=0, feedbackVersion=0, ceoFeedback=None) + self.events = [] + self.stale = False + self.fail_link = False + + def __call__(self, method, path, body=None): + self.events.append((method, path, body)) + if method == 'POST': + return self.row.copy() + if method == 'GET': + return self.row.copy() + if self.stale or (body.get('blogId') and self.fail_link): + raise Blocked('409 stale') + assert body['expectedVersion'] == self.row['version'] + assert body['feedbackVersion'] == self.row['feedbackVersion'] + self.row.update({k: v for k, v in body.items() if k not in ('expectedVersion', 'feedbackVersion')}) + self.row['version'] += 1 + return self.row.copy() + + +class ConsumerTest(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.api = API() + self.c = Consumer(Path(self.tmp.name) / 'journal.db', self.api) + self.created = [] + + def tearDown(self): + self.tmp.cleanup() + + def publish(self, draft): + self.created.append(draft) + return 'blog-id' + + def prepare(self): + return self.c.prepare('day-run', {'title': 'research'}) + + def review(self, packet, draft): + return dict(reviewer='independent-seo', directiveHash=digest(packet), draftHash=digest(draft), evidence='immutable://review-receipt') + + def test_research_before_draft_then_claim_create_link_idempotent(self): + packet = self.prepare() + draft = {'html': 'reviewed'} + result = self.c.publish('day-run', packet, draft, None, self.publish) + self.assertEqual(result, 'blog-id') + self.assertEqual(self.c.publish('day-run', packet, draft, None, self.publish), result) + self.assertEqual(len(self.created), 1) + events = self.api.events + self.assertEqual(events[0][0], 'POST') + self.assertEqual(events[-1][2]['blogId'], 'blog-id') + self.assertEqual([e for e in events if e[0] == 'PATCH'][0][2]['feedbackVersion'], 0) + + def test_directive_passed_and_requires_exact_review(self): + self.api.row.update(version=3, feedbackVersion=2, ceoFeedback='Focus on cats') + packet = self.prepare() + self.assertEqual(packet['ceoFeedback'], 'Focus on cats') + draft = {'html': 'cats'} + with self.assertRaises(Blocked): + self.c.publish('day-run', packet, draft, None, self.publish) + receipt = self.review(packet, draft) + receipt['draftHash'] = 'wrong' + with self.assertRaises(Blocked): + self.c.publish('day-run', packet, draft, receipt, self.publish) + self.c.publish('day-run', packet, draft, self.review(packet, draft), self.publish) + self.assertEqual(len(self.created), 1) + + def test_stale_claim_never_calls_publisher(self): + packet = self.prepare() + self.api.stale = True + with self.assertRaises(Blocked): + self.c.publish('day-run', packet, {}, None, self.publish) + self.assertEqual(self.created, []) + + def test_ambiguous_publish_blocks_restart(self): + packet = self.prepare() + def ambiguous(draft): + self.created.append(draft) + raise TimeoutError('unknown write outcome') + with self.assertRaises(TimeoutError): + self.c.publish('day-run', packet, {}, None, ambiguous) + restarted = Consumer(self.c.path, self.api) + with self.assertRaises(Blocked): + restarted.publish('day-run', packet, {}, None, self.publish) + self.assertEqual(len(self.created), 1) + + def test_link_failure_can_retry_link_only(self): + packet = self.prepare() + self.api.fail_link = True + with self.assertRaises(Blocked): + self.c.publish('day-run', packet, {}, None, self.publish) + self.api.fail_link = False + self.c.reconcile_link('day-run') + self.assertEqual(len(self.created), 1) + + def test_research_post_ambiguity_never_reposted(self): + def fail(*args): + raise TimeoutError('unknown') + c = Consumer(self.c.path, fail) + with self.assertRaises(TimeoutError): + c.prepare('run', {}) + with self.assertRaises(Blocked): + self.c.prepare('run', {}) + self.assertEqual(self.api.events, []) + + def test_packet_tampering_rejected(self): + packet = self.prepare() + packet['ceoFeedback'] = '' + with self.assertRaises(Blocked): + self.c.publish('day-run', packet, {}, None, self.publish) + + def test_actual_entrypoint_delivers_directive_before_create(self): + from daily_publisher import run_new + self.api.row.update(version=1, feedbackVersion=1, ceoFeedback='cats only') + order = [] + def draft(packet): + self.assertEqual(self.api.events[0][0], 'POST') + self.assertEqual(packet['ceoFeedback'], 'cats only') + order.append('draft') + return {'html': 'cats'} + def review(packet, article, directive_hash, draft_hash): + order.append('review') + return self.review(packet, article) + def create(article): + order.append('create') + return self.publish(article) + self.assertEqual(run_new(self.c, 'entry', {}, draft, review, create), 'blog-id') + self.assertEqual(order, ['draft', 'review', 'create']) + + def test_feedback_edit_during_draft_rejects_action_and_create(self): + from daily_publisher import run_new + def draft(packet): + self.api.row.update(version=1, feedbackVersion=1, ceoFeedback='new instruction') + self.api.stale = True + return {} + with self.assertRaises(Blocked): + run_new(self.c, 'entry', {}, draft, lambda *args: None, self.publish) + self.assertEqual(self.created, []) + + def test_feedback_after_claim_retains_created_identity_no_second_create(self): + packet = self.prepare() + def create(draft): + self.api.row.update(version=2, feedbackVersion=1, ceoFeedback='late') + return self.publish(draft) + with self.assertRaises(Blocked): + self.c.publish('day-run', packet, {}, None, create) + self.assertEqual(self.c.row('day-run')['blog'], 'blog-id') + with self.assertRaises(Blocked): + self.c.publish('day-run', packet, {}, None, self.publish) + self.assertEqual(len(self.created), 1) + + def test_overlapping_consumers_only_one_publish(self): + packet = self.prepare() + other = Consumer(self.c.path, self.api) + def nested(draft): + with self.assertRaises(Blocked): + other.publish('day-run', packet, draft, None, self.publish) + return self.publish(draft) + self.c.publish('day-run', packet, {}, None, nested) + self.assertEqual(len(self.created), 1) + + +if __name__ == '__main__': + unittest.main() diff --git a/specs/content-research-consumer/plan.md b/specs/content-research-consumer/plan.md new file mode 100644 index 00000000..c83bf1d7 --- /dev/null +++ b/specs/content-research-consumer/plan.md @@ -0,0 +1,6 @@ +# Plan +Use stdlib Python adapter under scripts/content-research, transport injection with real urllib authenticated implementation, SQLite durable write-ahead journal and callback boundary to existing authorized publisher. No credentials persisted. Backend contract v2 /opt/data/analytics-engine/contract.md verified directly against service actionFinding predicates at source df5584d. A PATCH claim protects the finding, not later blog creation. Do not invent a backend atomic blog transaction. + +Graph: existing graphify-out/graph.json SHA256 6a73473cee9e207543fdeeb668bea544f766200a1b16b8a3a9ce18d4e42e51af, binding unknown; inline vocabulary scan finds no publisher/seo-tracking nodes. It is navigation only. Direct source and runtime prompt inspections establish boundary. Initial worktree dirty .specify/, evidence/, graph cache belong to parent; do not stage. + +Activation requires separately authorized runtime provisioning and prompt/caller wiring, unchanged publisher ownership/QA wrappers, plus Stage acceptance. Adapter is deliberately not installed into active runtime. Cross-resource directive freshness through the actual blog transaction requires a future backend contract, not an adapter preflight. diff --git a/specs/content-research-consumer/spec.md b/specs/content-research-consumer/spec.md new file mode 100644 index 00000000..dcd16365 --- /dev/null +++ b/specs/content-research-consumer/spec.md @@ -0,0 +1,12 @@ +# Versioned daily-publisher research consumer + +Scope: dormant Git-owned integration adapter; no active runtime, cron, deployment or external writes. Daily publisher remains NEW-only and retains existing ownership, quota, backup, QA and ambiguous-write reconciliation. + +Verified consumer: `/opt/data/publishers/e4c9b52eb040/cron-prompt.txt` lines 11,13,17 instruct an agent to research, draft and call Admin create, with local publisher_state.py intent/record protection. Directory is outside Git (git rev-parse fails); neither this runtime nor `/opt/data/content-maintenance/engine` may be altered. No Git-owned daily creation module exists in current scripts inventory. + +Acceptance: +- Save a finding before drafting; persist request intent before sending, never blindly repeat ambiguous research creation. +- Fetch current finding and pass complete directive and exact version to drafting. CEO feedback is optional; non-empty feedback requires independent review evidence bound to exact directive and draft hashes. An evidence receipt cannot prove arbitrary free-text semantic compliance. +- Claim the exact finding/feedback version using Backend action endpoint before publishing; a 409 prevents create. A feedback edit after claim is not atomically excluded from blog creation by this API contract; disclose this gap, never claim cross-resource enforcement. +- Call existing publisher create callback at most once per journal; persist publication intent before callback and identity before linking. Ambiguity blocks retries. Link finding to returned blog only after creation; failed/stale link must retain ID and allow link-only reconciliation without another create. +- Existing publisher's safety tests remain unchanged. All new tests use isolated journals and explicit API/callback fixtures, not live writes. diff --git a/specs/content-research-consumer/tasks.md b/specs/content-research-consumer/tasks.md new file mode 100644 index 00000000..26b14215 --- /dev/null +++ b/specs/content-research-consumer/tasks.md @@ -0,0 +1,5 @@ +# Tasks +1. RED consumer-boundary regression tests. +2. Implement durable adapter and authenticated transport without runtime activation. +3. GREEN isolated tests, existing publisher safety regressions; document atomicity/activation gap. +4. Commit explicit owned files only; no push/deploy/publication.