diff --git a/research/p104-findings-nov15/NOTICE b/research/p104-findings-nov15/NOTICE new file mode 100644 index 0000000000..c4d0ac6aae --- /dev/null +++ b/research/p104-findings-nov15/NOTICE @@ -0,0 +1,10 @@ +Apache-2.0. DreamingOfClouds with Claude Opus 5.5 (Anthropic), via Claude Code. The outside-session lanes behind the DYINGPAIR, SLACK +and GAPLEMMA proofs and the native-side census scripts are DreamingOfClouds'. + +code/pricekit.py is our PRICEKIT, as shipped in #7607. data/w7546.json.gz is ToikY's #7546 word and data/7113_CALLER.json.gz is Rohan Arun's #7113 +signed caller, both as vendored in #7606/#7607 and used as data only. The frontier words that the lane scripts read (our #7615 and #7617; #7612 by +Clear1nsight and #7616 by SYLforge in their lineage) are used as data only and are not shipped. The funded-return and courier conventions come from +the crowd chain #7480/#7481; the retag/slide edit family goes back to #7277 and #7612. Credit CubeJerry for the #6805 native and the P104 price and +replay lineage. + +All checkers in this package are our own code; no code is imported from the repository or any other package, and no other author's open-PR code is run. diff --git a/research/p104-findings-nov15/PROOF.md b/research/p104-findings-nov15/PROOF.md new file mode 100644 index 0000000000..eeac12541f --- /dev/null +++ b/research/p104-findings-nov15/PROOF.md @@ -0,0 +1,198 @@ +# Proofs: dying roles, slack, and the Delta-Floor lemma (rho = 2 two-arm callers) + +Common setting and notation (as in the #7606 / #7609 caller sections): rho = 2, N = n + 2 units T = I - 2xy^T with product -I; +stages are adjacent isotropic unit pairs on coordinate masks Z_s with t_s != -1; prefix commutation; thin units arbitrary; two arms +meeting in a pair of hubs H. Tags: [P] proved here, [C] exact computation re-run by a checker in this package, [C-param] priced +with PRICEKIT, [N] numerical (float solver, not a claim). + +## Part I. DYINGPAIR + +### Theorem: tight dying roles are impossible for every interior death stage [P] + +Setting (rs2/rootcheck model, as NESTPROOF/DYINGGEN). rho = 2, N = n+2 units T = I - 2xy^T, product -I; stages = 2 adjacent isotropic units +in coordinate masks Z_s with t_s != -1; prefix commutation (P_u commutes with Pi_{Z_s} just before stage s); thin units arbitrary. +Two arms: left stages 0..m-1 (time order), right arm after the thin block; H = 2 hubs; some right stage r has Z_r meeting every left Z_j +only inside H. E_j = roles first appearing at left stage j; U_j = Z_0 u ... u Z_j. Normalised X~_a = X_a Z_a^-T, Y~_a = Y_a W_a^-T (K = WZ^T). +Layout: D (nonempty) is a subset of E_k, present in Z_k..Z_d, absent from Z_{K}, K = d+1, with d <= m-3 (so stage K+1 is still in the left arm). + +Theorem. Suppose (T1) |U_j| = 2j+4 for every k < j <= d (no extra role enters at stages k+1..d), and (T2) |Z_K| <= 2K+3 +(e.g. the B count 2K+4 minus the dying roles, plus at most |D|-1 extra roles). Then no such root exists. Covers singles and pairs, any +entry k, any death d <= m-3, one dying group per arm (the other arm's dying roles are outside every left mask, so do not enter). + +Facts used (all already proved): Lemma S (SINGLEHUB): G[later, earlier] = W Z^T, G[earlier, later] = -W Z^T; K_ss = [[1,-g],[h,1]], t = gh. +Floor (CALLEROPEN 3.2, step 2, which never uses nesting): rows Y_0..Y_{j+1} against columns X_0..X_j, X_r|H form a NONSINGULAR square +matrix of size 2j+4 (needs Z_i meet Z_r only in H for i <= j+1). Hence (F1) X_0..X_j, X_r|H are 2j+4 independent vectors of R^{U_j}; +(F2) the y's of stages 0..j+1 are independent (this is NESTPROOF's hypothesis H, now a consequence). + +Step 0 (Q has no eigenvalue 2). Q_a = I - Y~_a^T X~_a = W^-1 (K_aa - G_aa) K_aa^-1 W, and (K-G)K^-1 = [[1+2t,-g],[-h,1]]/(1+t): +trace 2, det 1/(1+t), eigenvalues 1 +- sqrt(t/(1+t)). An eigenvalue 2 forces det 0: impossible. [C qcheck.py: 1+-i at t=-1/2, exact match] +Step 1 (R^D-content is forced). Rows Y_j (j <= K+1) vs columns X_0..X_K, X_r|H: full matrix F nonsingular (Floor at K, needs K+1 <= m-1). +Restricting rows and columns to Z_K changes F by Y|_D^T X|_D (the only roles outside Z_K touched by these columns are D), so +2K+4 = rank F <= |Z_K| + rank(X_{= 1 (= |D| in the exactly tight case). +V := C(P_K - I) = C(X_{c). +With A_c = alpha_0+...+alpha_c and A_k = 0 this reads (2I - Q_c) A_c + Q_c A_{c-1} = 0. A_{-1} = 0 and Step 0 give A_0 = ... = A_{k-1} = 0, +so alpha_0 = ... = alpha_{k-1} = 0 and alpha_k = -A_{k-1} = 0: v = 0. Contradiction. QED +(k = 0 needs no Step 4; k = 1 is the identity Q_0 = 2I on Y~_0 for a pair.) + +Scope / what is NOT covered: (i) slack: an extra role entering at a stage in (k, d] (breaks T1), or >= |D| extra roles at the death stage K +(breaks T2, e.g. a single plus one extra role at K); (ii) one arm with dying roles of different entry stages; (iii) d = m-2 (death at the +innermost stage: no left stage K+1; DYINGGEN's conditional kill stands there); (iv) t = -1 stages, rho != 2, non-two-arm (tree) callers. + +## Part II. SLACK + +Notation as in Part I. + +A. True cap [P]. An idle line is ONE idle op covering stages a..b; at each stage k the lines alive give the complement of X_k in R^{Z_k} +with dual vectors (p ⟂ Y_k, q ⟂ X_k; CALLEROPEN cap_check.py computes |S| - max(rank Y|S, rank X|S); SINGLEHUB Lemma 5). One op has one +end-referred (p,q), so a line covering I=[a,b] has p,q in R^{Z_I}, Z_I = Z_a ∩..∩ Z_b, p ⟂ Y_a..Y_b, q ⟂ X_a..X_b: + N(I) <= |Z_I| - max(rank Y_I|Z_I, rank X_I|Z_I). +Per boundary (|I|=2) rank <= 4, so "|S|-4" is the TIGHTEST per-boundary cap, never too generous. The pricer's error is elsewhere: +layout_price.lines()/step1.capS impose caps only on |I|=2; for nested masks Z_I = Z_a so longer intervals add nothing (B reproduced +1.0950/1.0031/0.9773 by icap.py too), but a role that dies (or a lone extra role) inflates the two-stage caps while the long lines the +greedy keeps must live in the smaller Z_I. icap.py imposes every interval. +B. Extended Crossing Lemma [P]. With a witness (hubs ⊂ S'), rank(Y_k,Y_k+1|S') = rank(X_k,X_k+1|S') = 4 if ANY of X_k, X_k+1, Y_k, Y_k+1 +is supported in S'. Proof: CALLEROPEN 3.1 for X_k (K-G invertible); for X_k+1: c^T G_{k+1,k+1} = a^T G[k,k+1] = -c^T W_k+1 Z_k+1^T +gives c^T(G+K)=0 and G+K = [[1,0],[2h,1]] is invertible; Y-versions by transposition. Without one of these only rank >= 2 (witness) is proved. +C. Refined DYINGPAIR Step 3 [P]. If alpha_j != 0 is the top index (k= 2K+4-|Z_K|-(|D_all|-|D|)), Steps 2,4 unchanged. KILL: |Z_K|+|D_all|-|D| <= 2K+3 and no extra role entered at +stages <= d-1. This kills an extra role entering exactly at d (e.g. pair k=0 d=1 + role at 1, +0.75% in the rank-4 model). run3: 22/140 killed. +Lone extra role e on [je,ee]: by NESTPROOF L2 (X and Y sides) either X_{ B applies on every interval -> no gain, +or delta_e in span(X~_a, je<=a<=ee) with sum alpha = 0 (L1 + recursion for a 0 (NESTPROOF's open question), on BOTH sides, plus rank < 4 on the restricted sets. + +D. Rank drop => reflection block, with first-touch conditions [P] (lone e on [je,ee], K=ee+1, interval I=[a,b] across the death, Z_I = Z_a \ e, +normalised G~[c,j] = +I (jc); witness = right arm). + Y side: a relation of rank(Y_a,Y_a+1|Z_I) is w = Y~_a mu - Y~_a+1 mu (witness forces equal coefficients) supported on E_a+1 u {e}. Pairing with + X~_j: j 0 on both sides (the hypothesis is automatic). It also needs + X_a, Y_a to touch e while X_j, Y_j (j= K gives Phi = sum_{j<=t} gamma_j - gamma_r. + These 2(t-a+1)+2 functionals are independent on gamma-space (gamma_r only in Phi; then triangular with Q_s invertible). The domain + {gamma : c(gamma) in R^{Z_I}} has codim <= rank of span(X_<=t) on U_t \ Z_I = {e} u E_a+1 u .. u E_t. So rank(Y_I|Z_I) >= 2(t-a)+4 - codim. + (1) t = a, columns X_<=a: codim <= 1 (e row: sum_j x~_j gamma_j). A drop needs that constraint in span{Phi, Q_a gamma_a}, so x~_j = 0 for all j je. If t >= a+1: X~_t[E_t] alpha_t = 0 (E_t has no e), so X_t has rank <= 1 on E_t, and + codim <= 1 + 2(t-a-1) + 1 = 2(t-a), giving Y-rank >= 4. If t < a: the e row gives sum_{j<=t} x~_j alpha_j = 1, but (1) needs x~_j = 0 for j0 costs nothing at the Gram level; the obstruction is the coordinate support used above. +F. Pair + slack (the remaining survivors) [P partial]. If the pair D enters at k < a and |U_k| = 2k+4, then X~_k[D] is invertible (F1 at k). The + constraint functionals of D therefore always involve gamma_k, and the drop in (1) is <= #slack roles of Omega untouched by X_{= 3 there: the r=2 ceiling (+2.37%) is excluded, and the r=3 ceiling (+1.38%) stands. Intervals starting AT k (k>=1) are not covered. Not closed: the recursion + in E needs Y~_c^T v = 0 for c=1 with Z_I = U_{k-1}: rank(Y_I|Z_I) = rank(X_I|Z_I) = 2 EXACTLY. Every Y_s, s>=k, restricted to + U_{k-1} = W_s x (fixed 2 x (2k+2) matrix): the columns X_0..X_{k-1}, X_r|H span R^{U_{k-1}} (F1), and G = +-W_s Z^T. Dually for X. So + F's claim "rank-2 ceiling excluded" was wrong for k>=1. Conservative proved caps (run6.py: r = 4 - min(2,u), u = uncertified roles of + Omega_a) give a ceiling of +2.19% (c=2 k=1 d=3 slack 0..3). 74/117 survivors are >= +0.5% (run6.out). k=0 survivors stay <= +1.38%. + (2) Generalised kill [P, extends DYINGPAIR]: Delta = roles of U_d absent from Z_K. If all of Delta enters at one stage k', 2K+4-|Z_K| >= 1, + and |U_{t-1}| <= 2t+2 for k' 0. With b_K = 0 every interval has rank 4, and the whole menu prices <= +0.49% (run3/run4 'opt'). +GAP LEMMA (hand-off; the only open piece). rho=2 two-arm root (NESTPROOF model: prefix commutation, product -I, t!=-1), left arm stages + 0..m-1, B-count masks |U_j| = 2j+4 plus ONE extra (slack) role s entering at je <= d-1, and a dying set D (|D| = 1 or 2, entering k, alive k..d). + Delta = D u {s if s dies at K}, K = d+1 <= m-2. CLAIM: V_K cap R^Delta = 0, i.e. no nonzero v = sum_{j<=d} X~_j alpha_j with sum alpha = 0 + supported in Delta. Proved: Delta with a single entry stage and no slack entering in (k',d] [(2)]; lone s with no D [E: the rank stays 4 + even with b_K>0]. Open: (a) je <= k (s and D enter at different stages, or s enters before D); (b) k < je <= d-1 with top index t(v) in [je,d] + (Step 3 has one spare dimension, |U_{t-1}| = 2t+3); (c) k=0, s dying at K, t(v) = a (PROOF F). If the CLAIM holds, slack is closed at <= +0.49%. + If it fails, the best layout is pair k=1 d=3 + slack 0..3, with a ceiling of +2.19% [C-param, conservative caps]. The Block Rank Bound gives no kill (monotone). + Suggested attack: (b) via a second top index on the Y side (dual v), or an exact mod-p search for v in a B-type root with one extra role. + +## Part III. GAPLEMMA + +Notation as in Parts I and II. +Delta = roles of U_d absent from Z_K (K = d+1 <= m-2), Delta_i = Delta cap U_i, B = V_K cap R^Delta (b = dim B), +B_{<=i} = {v in B : alpha_j(v) = 0 for j > i}, W_i := V_{i+1} cap R^Delta. Facts used: F1/F2 (Floor, valid for j <= m-2), +L1 (sum zero), L2 (splitting of V_K at K), Step 0 (2I - Q_c invertible), the recursion phi_c := Y~_c^T v = (2I-Q_c)A_c + Q_c A_{c-1}. + +### P1. Delta-Floor Lemma [P] +For every 0 <= i <= d: (a) W_i = B_{<=i}; (b) dim B_{<=i} <= |Delta_{i-1}| (Delta_{-1} = empty); (c) dim B_{<=i} >= 2i+4 - |U_i \ Delta|. +Hence a root needs |Delta_{i-1}| >= 2i+4 - |U_i \ Delta| for every 0 <= i <= d. +Proof. (c) F1 at i: X_0..X_i, X_r|H are 2i+4 independent vectors; their span is S_i = V_{i+1} (+) R^H. Let pi delete the Delta +coordinates. If x + h lies in R^Delta (x in V_{i+1} <= V_K, h in R^H), split x by L2 at K: its R^{Z_K}-part is -h (H <= Z_K, +Delta <= N_K) and lies in V_K. F1 at d gives V_K cap R^H = 0, so h = 0. So ker(pi|S_i) = W_i and 2i+4 <= dim W_i + |U_i \ Delta|. +(a) x in W_i is supported off Z_K, so Y_K^T x = 0 and L1 gives sum-zero coefficients on stages <= i: x in B_{<=i}. The converse is clear. +(b) For x in B_{<=i}: A_i = 0. For c <= i-1, phi_c(x) = Y~_c[Delta cap Z_c]^T x only reads x on Delta_{i-1}. If x vanishes there, phi_c = 0 +for c <= i-1 and the recursion (2I - Q_c invertible, A_{-1} = 0) gives A_0 = .. = A_{i-1} = 0. With A_i = 0 this means every alpha_j = 0, so x = 0. QED +Consequences [P, dfloor.py]: (i) every layout in which all own roles of the outermost entry stage die (k=0) fails at i=0 (|U_0 \ Delta| <= 3). +(ii) every PAIR (c=2) layout fails: k=1 with s entering 0 and dying at K fails at i=1 (|Delta_0| = 1 < 6-4). k=1 with s surviving, or entering at 1, also fails at i=1. +So GAP-LEMMA open cases (a),(b),(c) are CLOSED for pairs, and for singles with k=0. dfloor.py kills 102 of the 117 run6 survivors. The 15 left are all +c=1 k=1: family A (slack 0..d dying, d=1..5, run6 ceilings +1.58..+2.17%) and family A' (slack surviving, <= +0.97%). +[N check, loc2.py / chk2.py] With Lemma S + supports + block-diagonal P_K + b=2 but WITHOUT F1, the c=2 k=1 d=2 slack 0..2 left arm solves (22/40). +Every solution with t != -1 violates F1 at j=2 (a left x-combination is supported on the hubs), as P1 predicts. So F1 (the right witness) is the essential input. + +### P2. Rank of the dying rows [P] +rank X_{= 1 needs x~_0s = 0 (Y side) or y~_0s = 0 (X side): rank 4. Only a = 0 may drop (to 3). + Priced with these proved caps (phantom_price.py): +0.37% (d=1), +0.23, +0.14, +0.08, +0.03% (d=5). All below +0.5%. +(b) If b = 2 (possible only for d >= 2, since b <= dim B_{<=1} + #{2<=t<=d : X~_t[E_t] singular} by the top-index filtration): R^Delta <= V_K, + so e_s and e_D lie in V_K with sum-zero coefficients, and e_D has alpha_0 = 0 (phi_0(e_D) = 0). NOT closed. This is the only + open piece of the >= +0.5% menu. (With thin units kept off Delta, b = 2 is forced, so family A d=1 dies and d >= 2 needs exactly this case.) + +### P4. Caps in the open case (family A, b = 2) [P, uses SLACK PROOF E(1)] +Across the death, Omega_a <= {s, D}. A one-step drop on [a,b] needs a combination c_s(s-row) + c_D(D-row) of the constraint functionals with +c_s x~_js + c_D x~_jD = 0 for all j < a (E(1)). For a >= 1, j = 0: x~_0D = 0 and x~_0s != 0 (P3), so c_s = 0 and only the D row can drop: rank >= 3. +For a = 0 only s is touched by X_0: rank >= 3. So r >= 3 on every interval (run6 used r = 2). Priced (phantom_price.py): +0.97 (d=1, but d=1 +cannot have b=2), +1.11 (d=2), +1.15 (d=3), +1.08 (d=4), +0.90% (d=5). Family A' (slack survives) stays <= +0.97% (run6 caps, already r = 3). +### P5. The open case is LOCALLY consistent [N, locgen.py A2b2, loc_A2b2_7.npy] +Left arm of family A d=2 (stages 0..2, 9 coords) with Lemma S, supports, x.y = 0, block-diagonal P_3 with P_3|Delta = -I_2, b = 2 sum-zero +vectors, F1 on the X and Y sides at j = 0,1,2 (with hub witness), and t = (-0.228, -0.775, -0.743): solved to 8.7e-15 (1 of 8 starts; the +b=1 spec A1 solved 1 of 4 with X[D] = Y[D] = 0, as P3(a) says). D is NOT a phantom there. So no argument confined to stages < K plus F1 can +close b = 2; a proof must use the stages >= K, the thin block or the right arm (e.g. Floor at K with rows Y_K, Y_K+1, or DYINGGEN Theorem 1 counts). +### P6. Extension round: reduction of the global question to the prefix M [P] + codimension [N] +Stages >= K, the right arm and (if b = |Delta|) the thin block never touch Delta, and P_K = diag(M, -I_Delta) with M = P_K|_{U_d \ Delta}. +So the total product is diag(Rest*M, -I_Delta*I) and every prefix commutation at stages >= K reads only Rest_partial*M. Lemma S is a +consequence of product -I, not an extra equation. Hence: a family-A root exists <=> (i) a left arm (3 stages for d=2) with P_K = diag(M,-I_2), +and (ii) a "rest" with Rest*M = -I, i.e. a root of the NON-dying layout L' = (B + one surviving extra role, one stage shorter) whose +first-two-stage prefix equals M. Since Delta is gone, L' has the same thin count. +[N codim.py] At loc_A2b2_7 the reachable set R = {M : diag(M,-I_2) = product of the masked left arm} has tangent dimension 33. The variety of +rank-4 prefixes on R^7 has dimension about 40, so R has codimension about 7. [N refac.py] A generic masked 2-stage M' is not refactorable (0/6 starts), which agrees with this. +So the open question is exactly whether the prefixes of L'-roots (a family whose dimension is unknown: no L' root has been built) meet R. +Not decided: no L' root exists in hand, the float solver fails at these sizes, and Groebner on the full system (hundreds of variables) is out of reach. + +### HANDOFF (standalone; parked Nov 15) +Status: the slack/dying-role prize in the rho=2 two-arm B caller is NOT closed. Proved ceiling +1.15% (family A d=3, b=2); everything else is <= +0.97%. +OPEN QUESTION (exact). Family A = a single role D entering left stage 1, plus a slack role s on stages 0..d, both dead at K = d+1 (d >= 2), with b_K = 2 +(R^{s,D} <= V_K). Let M = P_K restricted to U_d \ Delta (P_K = diag(M, -I_2)). Does some root of the non-dying layout L' (B + one surviving extra +role, one stage shorter, same thin count) have a first-two-stage prefix M' that lies in the reachable set R = {M : the masked 3-stage dying left arm +has P_K = diag(M,-I_2)}? R has codim ~7 among rank-4 prefixes on R^7 (d=2). Yes => family A exists (then price it, Block Rank Bound, PRICEKIT). +No => the caller closes at <= +0.97% (or <= +0.37% if A' also dies). +PROVED: P1 the Delta-Floor Lemma, which kills every pair layout and every k=0 layout (102/117 run6 survivors). P2 rank X[Delta] <= b_K, and b_K = |Delta| if thin units avoid Delta. + P3 family A with b=1 => D is a phantom, <= +0.37%. P4 with b=2, rank >= 3 on every interval, <= +1.15%. P6 reduction to the prefix M (the rest sees only M). +TRIED / WHY IT FAILED: (1) a left-arm-only proof is impossible: locgen.py A2b2 gives a local root (8.7e-15, F1 on both sides, t != -1, D non-phantom). + (2) refactoring generic masked 2-stage prefixes fails (refac.py 0/6), explained by the codim ~7 (codim.py: tangent dim 33). + (3) no L' root was available. The float solver fails at L >= 4-6, and Groebner on the full system (hundreds of variables) is infeasible. +NEXT TECHNIQUE (most promising): build an EXACT L' root (B + 1 extra surviving role) with the DYINGEXACT build.py/ctlB.py pipeline (Sigma1 chain -> + rank-2 splitting -> Schur block -> thin factorisation), extended by one role. Then compute the dimension of the L' prefix family at that root and + test transversality against R (rank of [dPrefix | T_R]). If the sum spans the ambient space, Newton within the joint system gives a root, which can + then be lifted mod p. For an obstruction instead: look for an invariant of M that every L' prefix satisfies (from Lemma S Gram with stage K, e.g. the + 2x2 E^T R~ data or Q_K) and that R violates. Compute it on loc_A2b2_7 vs a B-root prefix first. diff --git a/research/p104-findings-nov15/README.md b/research/p104-findings-nov15/README.md new file mode 100644 index 0000000000..fb868b21ec --- /dev/null +++ b/research/p104-findings-nov15/README.md @@ -0,0 +1,219 @@ +# P104 findings (Nov 15): caller dying-role proofs, native lock laws, spec ledger (no κ) + +This package collects the proofs and laws our lanes logged on Nov 15, after #7606, #7607 and #7609 (the follow-up findings package). It builds on +the P104 frontier words of the same day: our #7615 and #7617, whose lineage runs through Clear1nsight's #7612 (the 902/1091 rechart) and SYLforge's +#7616 (39 stationary auxiliary owned lifetimes). The funded-return/courier conventions used when pricing edits come from the crowd chain +#7480/#7481, and the chart-tag move family (retag plus slide) goes back to #7277's block recharts and #7612. The pricer is PRICEKIT (#7607). +Nothing already in #7606, #7607 or #7609 is repeated. Where a result extends one of them, it says so. + +It makes **no κ claim** and ships no word. + +Credit: DreamingOfClouds with Claude Opus 5.5 (Anthropic), via Claude Code. + +Tags: + +- **[P]** proved in [PROOF.md](PROOF.md) (or argued in the text, where stated). +- **[C]** exact computation, re-run by a checker in this package (see "How to verify"). +- **[C-param]** priced with PRICEKIT (`code/pricekit.py`, vendored from #7607; its control reproduces #7546's κ and S = 1424). +- **[C-lane]** exact computation by our lane code on a frontier word. The code is shipped under `code/`, but CI does not re-run it + (the words are other PRs' data, and some scripts need our lane's private helper modules; see "Lane scripts"). +- **[N]** numerical (float solver). Reported, never claimed. + +## Summary + +| # | result | tag | +|---|---|---| +| 1 | **DYINGPAIR.** No root of the ρ = 2 two-arm caller has a dying role pair (or single) entering left stage k and absent from stage d+1 (d ≤ nl−3), under the tight counts T1/T2. The Floor matrix is nonsingular without nesting; the triangular recursion has no eigenvalue 2. Kills every dying layout the pricer had marked as +1.1% to +2.4% | [P, C-param] | +| 2 | **SLACK.** The interval-capped line price (a line is one op, so N(I) ≤ \|Z_I\| − rank on every interval) corrects the pricer; nested B is unchanged, lone extra roles price ≤ 0 and are closed [P]; extended Crossing Lemma; any rank drop across a death forces b_K > 0 on both sides | [P, C-param] | +| 3 | **GAPLEMMA.** The Δ-Floor lemma (P1) kills every pair layout and every k = 0 layout (102 of the 117 survivors). 15 single-role layouts remain, with a proved ceiling of **+1.15%**; one open lemma (HANDOFF) decides them | [P, C, C-param] | +| 4 | **Native laws** on #7615/#7617: RELAY-DIAMOND, lattice lock, the dim-8 non-nesting (tails) law, and the three compiler laws JOINT-SUPPLY, FRAME-FANOUT and ROUTE-DEMAND-IS-UNIT-LEVEL | [P, C-lane] | +| 5 | **Bounds on the native.** Exact single-main reroute bound (= actual on 400/400 mains, forced D 1,811 of 3,950 D0/call), helper-layer floor 944 (1,366 with the endpoint census), zero-need relay census, and the D ledger by bucket | [P, C-lane, C-param] | +| 6 | **Spec ledger.** Which contract clauses are SPEC and which are verifier-only; new public registers are legal and a one-hop new helper costs 0 κ; the fresh-register "meet twice" (Sandwich) rule; S = 2·(104·16 − 17·56); mains are S-neutral | [P, C, C-param] | + +## How to verify + +``` +python3 -I -B research/p104-findings-nov15/code/gaplemma/dfloor_count.py # §3 Δ-Floor kill counts PASS_DFLOOR_COUNT +python3 -I -B research/p104-findings-nov15/code/slack/icap_check.py # §2 interval-cap correction PASS_ICAP +python3 -I -B research/p104-findings-nov15/code/dyingpair/step1.py # §1 priced menu, pass count PASS_DYINGPAIR_PRICE +python3 -I -B research/p104-findings-nov15/code/spec/shuttle.py # §6 one-hop helper is κ-free PASS_SHUTTLE +python3 scripts/research_package.py --package research/p104-findings-nov15 +``` + +All four checkers use the standard library plus the vendored `code/pricekit.py`. They read only `data/w7546.json.gz` (the #7546 word) and +`data/7113_CALLER.json.gz` (the #7113 caller), both as data, as vendored in #7606/#7607. Each has controls, listed in [VERIFY.json](VERIFY.json). + +--- + +## 1. DYINGPAIR: dying role pairs are impossible (PROOF Part I) + +Setting: ρ = 2 two-arm caller root (stages = pairs of adjacent isotropic units, t_s ≠ −1, prefix commutation, two arms meeting in the hubs H). +A set D of roles enters left stage k, is present on Z_k..Z_d, and is absent from Z_K with K = d+1 ≤ nl−2 (so stage K+1 is still in the left arm). + +**Theorem [P].** If (T1) no extra role enters at stages k+1..d, and (T2) |Z_K| ≤ 2K+3 (for example the B count 2K+4 minus the dying roles, plus at most +|D|−1 extra roles), then no root exists. This covers singles and pairs, any entry stage k, any death stage d ≤ nl−3, and one dying group per arm. + +Proof outline (full proof in PROOF Part I). The Floor matrix (rows Y_0..Y_{j+1} against columns X_0..X_j, X_r|H) is nonsingular **without** nesting, so +the nesting hypothesis H of #7609's NESTPROOF is automatic. Counting dimensions at K forces a nonzero D-supported vector v in C(P_K − I); Lemma S gives +sum-zero coefficients; the Floor dimension count puts its top index at k; and the triangular recursion (2I − Q_c)A_c + Q_c A_{c−1} = 0 kills it, because +Q_a has eigenvalues 1 ± √(t/(1+t)), never 2. + +What it kills: every dying layout in the pricer's menu with T1/T2: k = 0, d = 3 (priced +1.31%); k = 1, d = 4 (+1.08%); one pair per arm (+2.4%); and +the L = 6, nl = 4 stage-0 test (B_2 = 0). + +Scope. Not covered: an extra role entering inside (k, d] or at least |D| extra roles at the death stage (handled in §2 and §3); one arm with dying roles of +different entry stages; death at the innermost stage d = nl−2 (#7609's conditional kill stands there); t = −1 stages, ρ ≠ 2, non-two-arm callers. + +Checks. `step1.py` [C-param] reprices the 63 single-arm dying layouts and the per-arm pairs (B control 1.0950 / 1.0031 / 0.9773; 35 of 63 pass the +0.5% screen, +all now excluded by the theorem). `qcheck.py` (needs scipy through `rs2.py`; not run in CI) checks Step 0 on the exact nested L = 4 root: K has rank 2 and +Q_a has eigenvalues 1 ± i at t = −1/2, matching the prediction. `code/dyingpair/` also holds the Block Rank Bound scripts (`step2.py`, `o_s2_*.txt`). + +## 2. SLACK: pricing correction, lone roles, extended Crossing Lemma (PROOF Part II) + +| statement | tag | +|---|---| +| **A. True cap.** An idle line is one op covering stages a..b, so with I = [a, b] and Z_I = ∩Z: N(I) ≤ \|Z_I\| − max(rank Y_I\|Z_I, rank X_I\|Z_I). Per boundary the rank is ≤ 4, so "\|S\|−4" is the tightest per-boundary cap, never too generous. The pricer's error was elsewhere: it capped only two-stage lines, so a role that dies (or a lone extra role) inflated the two-stage caps while the long lines the greedy keeps must live in the smaller Z_I | [P] A | +| **Corrected prices.** With every interval capped at rank 4, nested B is unchanged (1.0950 / 1.0031 / 0.9773); "B plus one lone extra role" drops from +0.20..+0.59% to −0.03..−0.05%; the best pair+slack layout (k = 0, d = 4, slack 1..4) drops from +1.56% to +0.20% | [C-param] `icap_check.py` | +| **B. Extended Crossing Lemma.** With a witness, rank(Y_k, Y_{k+1}\|S′) = rank(X_k, X_{k+1}\|S′) = 4 if any of X_k, X_{k+1}, Y_k, Y_{k+1} is supported in S′ | [P] B | +| **C. Refined DYINGPAIR Step 3 / generalised kill.** An extra role entering exactly at d is killed; a whole dying set entering at one stage with no earlier slack role is impossible (Step 1 on Δ, top index, recursion at the entry stage) | [P] C, H.2 | +| **D. Rank drop ⇒ reflection block.** A rank drop of Y_I\|Z_I or X_I\|Z_I across a death forces b_K = dim(V_K ∩ R^Δ) > 0 on **both** sides, and forces X_a, Y_a to touch the extra role while X_j, Y_j (j < a) do not | [P] D, H.3 | +| **E. Lone extra roles are closed.** For a lone role e on [j_e, e_e], max(rank_X, rank_Y) = 4 on every interval across its death, for any window length (even with b_K > 0). Every lone role prices ≤ 0 | [P] E | +| **F/H. Pair + slack.** For a pair entering at k ≥ 1, intervals starting at k have rank exactly 2 on U_{k−1}, so the proved-cap ceiling was +2.19% (pair k = 1, d = 3, slack 0..3; 74 of 117 survivors ≥ +0.5%). This is the number §3 then cuts to +1.15% | [P] F, H | +| Side check: for a two-stage window block, every sum-zero vector of the window x's is a −1 eigenvector of the block, so b_K > 0 costs nothing at the Gram level; the obstruction is the coordinate support | [C] `win.py` (sympy; not run in CI) | + +Scope. ρ = 2, t ≠ −1, two-arm roots in the NESTPROOF model (prefix commutation, product −I). Numbers are model prices [C-param]; no root of any slack layout has been built. +The float solver fails even the lone-role control at L = 4, so no counterexample exists either. + +## 3. GAPLEMMA: the Δ-Floor lemma and the standalone open question (PROOF Part III) + +Notation: Δ = roles of U_d absent from Z_K (K = d+1 ≤ nl−2), B = V_K ∩ R^Δ, b = dim B. + +| statement | tag | +|---|---| +| **P1. Δ-Floor Lemma.** For every 0 ≤ i ≤ d: W_i = B_{≤i}, dim B_{≤i} ≤ \|Δ_{i−1}\|, and dim B_{≤i} ≥ 2i+4 − \|U_i \ Δ\|. So a root needs \|Δ_{i−1}\| ≥ 2i+4 − \|U_i \ Δ\| for every i. Corollaries: every layout whose outermost entry stage dies completely (k = 0) fails at i = 0; every pair (c = 2) layout fails at i = 1 | [P] P1, [C] `dfloor_count.py` | +| **Menu count.** Of the 140 pair/single + slack layouts, 117 survive the generalised kill; P1 kills 102 of them (all pairs, all k = 0). The 15 left are all c = 1, k = 1 | [C] `dfloor_count.py` | +| **P2.** rank X_{ 0 on both sides; an unconditional nesting theorem would close the slack class. +3. Is there a token-flow lower bound for the helper layer (per-frame span equality on full contents, including helper tokens) above the sound 944/1,366 floor, or a tightness theorem? The relay and zero-need census locates 4,129 D0/call at zero-need frames. +4. How much of the 1,811-of-3,950 D0/call forced mains gap (B2) is forced by anisotropy once helper tokens are counted (a joint mains+helper bound)? + +## Lane scripts (not re-run in CI) + +`code/relay`, `code/jointedit`, `code/tails`, `code/mainsgap`, `code/helperlb`, `code/dledger`, `code/netc9`, `code/netc17`, `code/netc19` and `code/spec` (except `shuttle.py`) are our lane code, with absolute paths rewritten +to `inputs/...`. They read frontier words as data: the word of #7615 (`inputs/ATTACK/COURIER226/hit_c226_1.json.gz`), of #7617 (`inputs/ATTACK/STACK7616/best7616.json.gz`), of #7608 (`w7608.json.gz`), of #7616 (`pr7616_word.json.gz`), the #7113 caller +(`inputs/MONOCALLER/7113_CALLER.json`, the vendored `data/7113_CALLER.json.gz` decompressed) and PRICEKIT (`inputs/PRICEKIT/pricekit.py`, a copy of `code/pricekit.py`). Some scripts also import our private lane modules (NETC6/NETC7 compilers, DCUT, SEEDBUILD) that are not shipped. +They are provided so every number above can be traced to code; the claims that CI re-runs are exactly those of "How to verify". Third-party words are used as data only; no other author's code is run. diff --git a/research/p104-findings-nov15/SOURCES.json b/research/p104-findings-nov15/SOURCES.json new file mode 100644 index 0000000000..ac8b60528f --- /dev/null +++ b/research/p104-findings-nov15/SOURCES.json @@ -0,0 +1,12 @@ +{ + "schema_version": 1, + "base_commit": "e54523517d0f03e58352e647c403a5af3e525a57", + "files": [ + { + "path": "docs/research/agent-task-catalog.json", + "usage": "provenance", + "git_blob": "3404fbdd5e7529a0674eef26114b5071e4586cec", + "sha256": "ead6cdc9d0af6d58e8a49744d8ec8c7c15e13d2ba93c276c434bcf9b51b93a81" + } + ] +} diff --git a/research/p104-findings-nov15/TASK.json b/research/p104-findings-nov15/TASK.json new file mode 100644 index 0000000000..c1982b313a --- /dev/null +++ b/research/p104-findings-nov15/TASK.json @@ -0,0 +1,135 @@ +{ + "schema_version": 1, + "task_id": "p104-side-correction", + "priority_revision": 8, + "base_commit": "e54523517d0f03e58352e647c403a5af3e525a57", + "original_card": { + "id": "p104-side-correction", + "question": "Can joint helper bases and a legal causal split improve the complete P104 productive/paid invoice?", + "starting_points": [ + "docs/research/current-priorities.md", + "docs/research/research-family-status.md", + "certificates/selected-result.json", + "docs/research/pr-intake-evidence/pr5599-completion/PROOF-REVIEW.md", + "research/p104-helper-subspace-splits/README.md", + "research/dyadic-helper-channel-capacity/README.md" + ], + "methods": [ + "Construct and independently check one source-bound instance of the stated first experiment.", + "Seek an exact counterexample or target-specific complete-budget obstruction to that same proposed change." + ], + "acceptance_gate": "An exact arbitrary-input identity, source/target frame availability and restoration at the chosen cut, plus the complete channel+correction+idle/hosting bill or an explicit favorable bound. The native channel alone is not a supplier. A scoped local result may be retained without a complete supplier. Expanding it into further construction rounds requires the family-level budget and obligation gate in research-family-status.md; an optimistic passing relaxation alone is not acceptance.", + "first_step": { + "command": [ + "python3", + "-B", + "scripts/research_selected_baseline.py" + ], + "expected": "PASS_RESEARCH_BASELINE_BINDINGS_ONLY", + "inspect": "Read revision 8, the selected review and live exact-source handoffs. Then follow this card's bounded first_experiment.", + "scope": "Source-bound orientation only, not a supplier replay or theorem proof." + }, + "first_experiment": "Start with the selected #5599 emitted word and review pending #5718 distinct input/output bases plus #5717/#5719 coverage before claiming work. Pick one uncovered source word, helper group and cut region. Prove the arbitrary-input relation and legal entry/exit corrections, then compare full target and paid profiles. Reject if completion, owned lifetimes or consumer costs repay the gain. A static pairing rank alone is not a physical split.", + "bottleneck_link": "Judge the actual current supplier contract and complete signed target/paid bill; frozen-profile ceilings require an applicability proof.", + "allowed_variations": [ + "One exact source head, helper/cut region or quantified proposition and changed hypothesis; check live handoffs first. Different seeds are not different scopes." + ], + "scope_change_triggers": [ + "Changing source head, helper/cut domain, operation alphabet or caller boundary without recording the new scope.", + "Treating an archived proposal or local numerical pass as complete supplier admission." + ], + "overlap_terms": [ + "P104", + "helper basis", + "helper split", + "5718", + "dyadic capacity" + ], + "existing_checks": [ + { + "scope": "PR #4133: literal-slot defect", + "evidence": "research/p104-literal-slot-defect/README.md", + "remaining": "Independently validate the stated scope or change its producer/consumer premise and pay the complete boundary.", + "status": "author-checked", + "source_commit": "88666f1dee93fbd85b31fd5d6f91a219e111b5f4", + "selected_record_sha256": "bb2eaa58647b353f88d0aec2eafb9d30df1be53589231b23e12ceef3e4bad129" + }, + { + "scope": "PR #4136: uniform commutation gap", + "evidence": "research/p104-uniform-commutation-gap/README.md", + "remaining": "Independently validate the stated scope or change its producer/consumer premise and pay the complete boundary.", + "status": "author-checked", + "source_commit": "281ce601a8be9db470de630759179c23e1dfaa88", + "selected_record_sha256": "bb2eaa58647b353f88d0aec2eafb9d30df1be53589231b23e12ceef3e4bad129" + }, + { + "scope": "PR #4140: uniform encoding gap", + "evidence": "research/p104-uniform-encoding-gap/README.md", + "remaining": "Independently validate the stated scope or change its producer/consumer premise and pay the complete boundary.", + "status": "author-checked", + "source_commit": "de25c6525aa866962309df5d107b32d55e6e5628", + "selected_record_sha256": "bb2eaa58647b353f88d0aec2eafb9d30df1be53589231b23e12ceef3e4bad129" + }, + { + "scope": "PR #4153: source-chart shortcut exhausts its optimistic allowance", + "evidence": "research/p104-source-chart-shortcut/README.md", + "remaining": "Independently validate the stated scope or change its producer/consumer premise and pay the complete boundary.", + "status": "author-checked", + "source_commit": "9123241fea53df77e51ffe30d9aca1625d4bf3a8", + "selected_record_sha256": "bb2eaa58647b353f88d0aec2eafb9d30df1be53589231b23e12ceef3e4bad129" + }, + { + "scope": "PR #4154: fiber-tag seam mismatch", + "evidence": "research/p104-fiber-tag-seam/README.md", + "remaining": "Independently validate the stated scope or change its producer/consumer premise and pay the complete boundary.", + "status": "author-checked", + "source_commit": "4ffcdaee5d6255a9cea1fe53a5424435068089d0", + "selected_record_sha256": "bb2eaa58647b353f88d0aec2eafb9d30df1be53589231b23e12ceef3e4bad129" + }, + { + "scope": "Selected PR5599: complete helper-subspace construction and independent caller replay; five framework assumptions remain.", + "evidence": "docs/research/pr-intake-evidence/pr5599-completion/PROOF-REVIEW.md", + "remaining": "The next experiment must identify uncovered source-specific work. Successor archive intake does not transfer this admission.", + "status": "reviewed", + "source_commit": "09dd0965711ca7eeb169057a420abab1c9052137", + "selected_record_sha256": "2a0220e866417774606e0f991e41b9c1d26b1f43b97857238cf478dbc7d330b5" + } + ] + }, + "scope_status": "changed", + "actual_scope": "Findings of Nov 15 following #7606/#7607/#7609 (no kappa, no word): (1) DYINGPAIR theorem: no dying role pair or single in the rho = 2 two-arm caller under counts T1/T2 (Floor matrix nonsingular without nesting, triangular recursion with Q_a eigenvalues 1 +- sqrt(t/(1+t)) != 2); (2) SLACK: interval-capped line pricing correction, extended Crossing Lemma, lone extra roles closed, rank drop forces b_K > 0 on both sides; (3) GAPLEMMA: Delta-Floor lemma kills every pair and k = 0 layout (102 of 117), proved ceiling +1.15% for the 15 single-role layouts, standalone open question; (4) native laws on the #7615/#7617 words: RELAY-DIAMOND, lattice lock, dim-8 non-nesting tails law, JOINT-SUPPLY / FRAME-FANOUT / ROUTE-DEMAND-IS-UNIT-LEVEL; (5) exact single-main reroute bound and forced D, helper-layer lower bound 944 / 1,366 and zero-need relay census, D ledger by bucket; (6) spec ledger: SPEC vs verifier clauses, new public registers legal with one-hop helpers free in kappa, the fresh-register meet-twice (Sandwich) rule, S = 2*(104*16 - 17*56), mains S-neutral.", + "scope_change_reason": "The card asks for a constructive helper-basis or causal-split improvement. This package records theorems, scoped laws and bounds on P104 (no kappa claim, no word) so that others can see which caller layouts, native edits and contract relaxations are closed or bounded, and why.", + "bottleneck_link": "The P104 invoice's fixed costs (the caller header and per-port odd unit, the owned decoder mass, the native helper layer's 9,962 D0 per call) are what every proposed redesign has hit; this package proves where the caller part is closed (DYINGPAIR, SLACK, GAPLEMMA), bounds the native part (lock laws, helper floor, mains bound) and records which contract clauses are SPEC.", + "budget_minutes": 240, + "stop_rule": "Every proved claim has a proof in PROOF.md or a stated argument; every finite fact tagged [C] is re-run by a checker with controls; lane-only numbers are tagged [C-lane] or [C-param]; open items stated as open; validator PASS.", + "outcome": "negative-result", + "evidence": "README.md, PROOF.md, code/ (four CI checkers: gaplemma/dfloor_count.py, slack/icap_check.py, dyingpair/step1.py, spec/shuttle.py; lane scripts), data/, VERIFY.json.", + "author": "DreamingOfClouds", + "ai_assistance": "Claude Opus 5.5 (Anthropic), via Claude Code.", + "open_questions": [ + { + "question": "Does a root of the non-dying layout L' (B plus one surviving extra role, one stage shorter) have a first-two-stage prefix in the set reachable by the dying family-A left arm (codimension about 7)?", + "why_it_matters": "It decides whether the single-role dying layouts exist (proved ceiling +1.15%) or the caller closes at <= +0.97%.", + "first_step": "Build an exact L' root with the exact-root pipeline, compute its prefix family dimension and test transversality against the reachable set.", + "blocked_by": "Nothing." + }, + { + "question": "Are interior stages of two-arm callers nested (b_k = 0 for 1 < k < nl-1)?", + "why_it_matters": "Any rank drop across a death needs b_K > 0 on both sides, so nesting would close the slack class.", + "first_step": "Exact two-arm search with one dying role at stage 2, extending the DYINGPAIR recursion.", + "blocked_by": "Nothing." + }, + { + "question": "Is there a token-flow lower bound for the helper layer above the sound 944 / 1,366 D0 per call floor?", + "why_it_matters": "4,129 D0 per call of the native sit at zero-need relay frames that no rank or contact bound sees.", + "first_step": "State per-frame span equality on full contents including helper tokens, with the row contract fixing which tokens reach which finals, and solve it as an LP.", + "blocked_by": "Nothing." + }, + { + "question": "How much of the 1,811 of 3,950 D0 per call forced mains gap is forced by anisotropy once helper tokens are counted?", + "why_it_matters": "It separates the D-side ceiling of x1.209 into a forced part and a design part.", + "first_step": "Lower-bound relay contacts per X-Y transfer under the Sandwich rule and price them against the mains gap.", + "blocked_by": "Nothing." + } + ] +} diff --git a/research/p104-findings-nov15/VERIFY.json b/research/p104-findings-nov15/VERIFY.json new file mode 100644 index 0000000000..8435f159d1 --- /dev/null +++ b/research/p104-findings-nov15/VERIFY.json @@ -0,0 +1,49 @@ +{ + "schema_version": 1, + "status": "ready", + "python": "3.12", + "requirements": [], + "inputs": [], + "commands": [ + { + "argv": [ + "{python}", + "-I", + "-B", + "{package}/code/gaplemma/dfloor_count.py" + ], + "timeout_seconds": 60, + "checks": "Stdlib. Enumerates the 140 pair/single + slack layouts of the L = 17, nl = 8 two-arm B caller; 117 survive the refined DYINGPAIR / generalised kill; the Delta-Floor criterion kills 102; the 15 left are all c = 1, k = 1 (no pair survives). Controls: plain B has no dying set; a k = 0 single is killed at i = 0; family A (k = 1, d = 3, slack 0..3) is not killed by P1. Prints PASS_DFLOOR_COUNT." + }, + { + "argv": [ + "{python}", + "-I", + "-B", + "{package}/code/slack/icap_check.py" + ], + "timeout_seconds": 120, + "checks": "PRICEKIT on data/w7546.json.gz and data/7113_CALLER.json.gz. Interval-capped line pricing: nested B control unchanged (1.0950 / 1.0031 / 0.9773) under old boundary caps and interval caps; B plus a lone extra role (1..3, 0..4, 2..5, 3..6) prices > +0.15% with boundary caps and <= 0 with interval caps; pair k = 0 d = 4 + slack 1..4 goes +1.56% -> +0.20%. Prints PASS_ICAP." + }, + { + "argv": [ + "{python}", + "-I", + "-B", + "{package}/code/dyingpair/step1.py" + ], + "timeout_seconds": 400, + "checks": "PRICEKIT. Reprices the two-arm B layouts with dying single/pair roles: B control 1.0950 / 1.0031 / 0.9773; 35 of 63 single-arm layouts pass the +0.5% screen (all excluded by the DYINGPAIR theorem); per-arm pair rows. Prints PASS_DYINGPAIR_PRICE." + }, + { + "argv": [ + "{python}", + "-I", + "-B", + "{package}/code/spec/shuttle.py" + ], + "timeout_seconds": 300, + "checks": "PRICEKIT, exact Decimal. On the #7546 word: a new 1-hop public register 4 -> 9 replays exactly and leaves kappa equal to the base to 34 digits (S, D0 unchanged); the 0 -> 4 -> 9 helper raises D0 by exactly 17 (f(9) - f(4) - f(5)) = 105.105 and keeps S. Prints PASS_SHUTTLE." + } + ] +} diff --git a/research/p104-findings-nov15/code/dledger/LEDGER.tsv b/research/p104-findings-nov15/code/dledger/LEDGER.tsv new file mode 100644 index 0000000000..21a07c9f96 --- /dev/null +++ b/research/p104-findings-nov15/code/dledger/LEDGER.tsv @@ -0,0 +1,28 @@ +bucket D0_per_call_7615 D0_per_call_7616 delta +C.T arm interior (deg2..7/8) A 21528.81 21528.81 0.00 +N.T helper birth-dim 0 13876.80 13876.80 0.00 +C.T hub (deg17) A 7612.66 7612.66 0.00 +C.T arm top A 6601.85 6601.85 0.00 +C.T arm interior (deg2..7/8) B 1794.07 1794.07 0.00 +N.T helper birth-dim 5 677.90 677.90 0.00 +C.T hub (deg17) B 634.39 634.39 0.00 +C.T arm top B 550.15 550.15 0.00 +C.T single (deg1) A 446.68 446.68 0.00 +N.T helper birth-dim 3 298.31 298.31 0.00 +N.T helper birth-dim 4 241.42 241.42 0.00 +N.T helper birth-dim 2 177.08 177.08 0.00 +N.T helper birth-dim 1 175.48 175.48 0.00 +N.T helper birth-dim 6 49.44 49.44 0.00 +C.T single (deg1) B 37.22 37.22 0.00 +N.T helper birth-dim 7 13.86 13.86 0.00 +N.T helper birth-dim 8 0.00 0.00 0.00 +C.P thin QUERY A 0.00 0.00 0.00 +C.P idle line (ODD) B -37.22 -37.22 0.00 +N.P owned reg -118.00 -118.00 0.00 +N.P main X2 -611.31 -611.31 0.00 +N.P main X -662.16 -664.41 -2.25 +N.P main Y2 -670.16 -667.66 2.50 +N.P main Y -737.01 -760.76 -23.76 +C.P idle line B -2004.36 -2004.36 0.00 +N.P helper MOVE -5548.36 -5537.88 10.49 +C.P idle line A -21234.08 -21234.08 0.00 diff --git a/research/p104-findings-nov15/code/dledger/ledger.py b/research/p104-findings-nov15/code/dledger/ledger.py new file mode 100644 index 0000000000..f3f6670389 --- /dev/null +++ b/research/p104-findings-nov15/code/dledger/ledger.py @@ -0,0 +1,70 @@ +import sys, math, json +sys.path.insert(0,'inputs/PRICEKIT') +import pricekit as pk +from collections import Counter, defaultdict +f=lambda r: r*math.log(r) +CALLER='inputs/MONOCALLER/7113_CALLER.json' +WORDS={'7615':'inputs/ATTACK/COURIER226/hit_c226_1.json.gz', + '7616':'inputs/ATTACK/WATCH/work/pr7616_word.json.gz'} +C=pk.load(CALLER); ports=pk.caller_ports(C); NST=C['stages'] +def ledger(wpath): + W=pk.load(wpath); nat=pk.native_from_word(W); ci=pk.caller_invoice(C) + L=ci['nJ']; inv=pk.invoice(ci,nat); pr=pk.price(inv,30) + B=defaultdict(lambda:[Counter(),Counter()]) # bucket -> [T,P] histograms (totals over all calls/ports) + # caller T: fields + for ty,t in C['types'].items(): + for dg in t['diagonal']: + bits=dg[2:]; d=sum(bits); r=int(dg[0])%2+8*d + idx=[k+1 for k,b in enumerate(bits) if b] + if d==NST: name='C.T hub (deg17)' + elif d==1: name='C.T single (deg1)' + elif idx[0]==1 and d==8 or idx[-1]==NST and d==9 and idx[0]==9 or (idx[0]==1 and d==8): name='C.T arm top' + elif idx[-1]==NST and idx[0]==9 and d==9: name='C.T arm top' + else: name='C.T arm interior (deg2..7/8)' + B[name+' %s'%ty][0][r]+=ports[ty] + # caller P + for e in C['events']: + if e[0]=='J': continue + ty,op=e[0],e[1]; k=ports[ty] + if op=='QUERY': B['C.P thin QUERY '+ty][1][1]+=k + elif op=='IDLE_ONE': B['C.P idle line '+ty][1][8*len(e[3])]+=k + elif op=='IDLE_ODD': B['C.P idle line (ODD) '+ty][1][8*len(e[3])+1]+=k + # J/host ops: ADD/SWAP/SCALE/HOST/LAMBDA unpriced + n=len(W['initial']); live=set(W['live']); H=W.get('hosted') or {} + X2,Y2=set(H.get('X2',[])),set(H.get('Y2',[])) + cls={} + for r in live: cls[r]='X'; cls[r+112]='Y' + for r in X2: cls[r]='X2' + for r in Y2: cls[r]='Y2' + dim=[len(b) for b in W['frames']] + for e in W['program']: + if e[0]=='MOVE': + rk=abs(dim[e[3]]-dim[e[2]]); r=e[1] + if r in cls: nm='N.P main '+cls[r] + elif r>=n: nm='N.P owned reg' + else: nm='N.P helper MOVE' + B[nm][1][rk]+=L + main=live|{r+112 for r in live}|X2|Y2 + for r in range(n): + if r in main: continue + d0,d1=dim[W['initial'][r]],dim[W['final'][r]] + B['N.T helper birth-dim %d'%d0][0][d1-d0]+=L + return pr,B,L,inv +def Dof(T,P): return sum(v*f(r) for r,v in T.items())-sum(v*f(r) for r,v in P.items()) +def Sof(T,P): return sum(r*v for r,v in T.items())-sum(r*v for r,v in P.items()) +out={} +for lab,wp in WORDS.items(): + pr,B,L,inv=ledger(wp) + Tt=Counter(); Pt=Counter() + for T,P in B.values(): Tt.update(T); Pt.update(P) + exact = (+Tt==+inv['T']) and (+Pt==+inv['P']) + rows=sorted(((Dof(T,P),Sof(T,P),b,sum(T.values()),sum(P.values())) for b,(T,P) in B.items()),reverse=True) + out[lab]=dict(kappa=pk.kstr(pr['kappa'],14),S=int(pr['S']),D0=float(pr['D0']),exact_hist=exact,sumD=sum(r[0] for r in rows),L=L, + rows=[dict(D=r[0],S=r[1],b=r[2],nT=r[3],nP=r[4]) for r in rows], + detail={b:dict(T=dict(sorted(T.items())),P=dict(sorted(P.items()))) for b,(T,P) in B.items()}) + print(lab,out[lab]['kappa'],out[lab]['S'],out[lab]['D0'],'hist-exact',exact,'sumD',out[lab]['sumD'],'L',L) +json.dump(out,open('ledger.json','w'),default=str) +a,b=out['7615'],out['7616']; bb={r['b']:r['D'] for r in b['rows']} +print('%-40s %12s %12s %10s'%('bucket','D0/call 7615','D0 7616','delta')) +for r in a['rows']: + print('%-40s %12.2f %12.2f %10.2f'%(r['b'],r['D']/17,bb.get(r['b'],0)/17,(bb.get(r['b'],0)-r['D'])/17)) diff --git a/research/p104-findings-nov15/code/dyingpair/cap_check.py b/research/p104-findings-nov15/code/dyingpair/cap_check.py new file mode 100644 index 0000000000..725a1d0da2 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/cap_check.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python3 +"""CALLEROPEN cap_check.py [C exact mod p]: on the BUILT #7113 B program (and joint_mono A as rho=1 contrast) compute, per call k, +the end-referred x/y spans (sl_lib = copy of CALLEROUT/sl_check.py) and check: + (1) Lemma C: rank(Y_k, Y_k+1 restricted to S = Z_k cap Z_k+1) and rank(X_k, X_k+1 | S) -> crossing cap |S| - rank; + (2) Floor lemma: rank(X of left calls <= k, plus the hub coordinate vectors) = 2(k+1) + 2 <= |Z_k| (and right arm mirrored); + (3) the B idle crossings actually used (from the IDLE runs) vs the cap. Random D, q0 = 1 and i.""" +import json, random, sys +from sl_lib import * +def spans(C, part, q0, seed=7): + n = C['roles']; rnd = random.Random(seed); Dv = {s: rnd.randrange(2, P) for s in range(1, 18)} + R = ident(n); snaps = {}; order = [] + for e in C['events']: + if e[0] == 'J': + apply_native(R, C, e, q0, Dv, 'A'); apply_native(R, C, e, q0, Dv, 'B'); snaps[e[1]] = (e, [r[:] for r in R]); order.append(e[1]) + else: apply_op(R, e, q0, Dv) + F = R; T = C['types'][part]; off = T['offset']; m = len(T['diagonal']) + masks = {k: {off + i for i, dg in enumerate(T['diagonal']) if dg[1 + k]} for k in range(1, 18)} + XS = {}; YS = {} + for k, (e, Rk) in snaps.items(): + Jp = ident(n); apply_native(Jp, C, e, q0, Dv, part); Jm = ident(n); apply_native(Jm, C, e, (P - q0) % P, Dv, part) + s = matmul(Jp, inv(Jm)); S = matmul(F, inv(Rk)); E = [[(s[i][j] - (i == j)) % P for j in range(n)] for i in range(n)] + Ee = matmul(matmul(S, E), inv(S)) + XS[k] = basis([list(c) for c in zip(*Ee)]); YS[k] = basis(Ee) + return order, masks, XS, YS, off, m +def restrict(vecs, S): return [[v[i] for i in sorted(S)] for v in vecs] +for fn, part in (('../MONOCALLER/7113_CALLER.json', 'B'), ('../MONOCALLER/joint_mono.json', 'A')): + C = json.load(open(fn)) + for q0 in (1, IM): + order, Z, XS, YS, off, m = spans(C, part, q0) + hubs = [r for r in range(off, off + m) if all(r in Z[k] for k in range(1, 18))] + print('%s part %s q0=%s: call order %s; hubs %s; |Z| %s' % (fn.split('/')[-1], part, 'i' if q0 == IM else q0, order == sorted(order), [h - off for h in hubs], [len(Z[k]) for k in order])) + line = [] + for a, b in zip(order, order[1:]): + S = Z[a] & Z[b] + ry = rank(restrict(YS[a] + YS[b], S)); rx = rank(restrict(XS[a] + XS[b], S)) + line.append('%d|%d:S%d ry%d rx%d cap%d' % (a, b, len(S), ry, rx, len(S) - max(rx, ry))) + print(' boundaries:', ', '.join(line)) + # floor lemma: left arm = calls whose mask grows; use first half where |Z| increases + sizes = [len(Z[k]) for k in order]; peak = sizes.index(max(sizes)) + fl = [] + for i in range(len(order)): + k = order[i]; A = order[:i + 1] if i <= peak else order[i:] + vec = sum((XS[j] for j in A), []) if i <= peak else sum((YS[j] for j in A), []) + hv = [[int(c == h) for c in range(len(vec[0]))] for h in hubs] + fl.append('%d:%d/%d/%d' % (k, rank(vec), rank(vec + hv), len(Z[k]))) + print(' floor (rank X_arm<=k / +hubs / |Z_k|):', ' '.join(fl), flush=True) diff --git a/research/p104-findings-nov15/code/dyingpair/chk.py b/research/p104-findings-nov15/code/dyingpair/chk.py new file mode 100644 index 0000000000..3ca04d8e08 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/chk.py @@ -0,0 +1,30 @@ +"""ARMTREE chk.py: check the Block Rank Bound (ilp2) on a mask layout; layouts for L=8: B form, toy 'two bounded headers'.""" +import sys, itertools, numpy as np +from ilp2 import samples, minrank +def masks_B(L,nl,hubs=2,grow=2): + ms=[set(range(L))]*hubs + for k in range(nl): ms+= [set(range(k,nl))]*grow + for k in range(L-1,nl-1,-1): ms+=[set(range(nl,k+1))]*grow + return ms +def piece(lo,hi,nl,grow=2): # time-contiguous mountain piece stages lo..hi-1, centre between lo+nl-1 | lo+nl + ms=[] + for k in range(lo,lo+nl): ms+=[set(range(k,lo+nl))]*grow + for k in range(hi-1,lo+nl-1,-1): ms+=[set(range(lo+nl,k+1))]*grow + return ms +def toy(L=8,hubs=2,npieces=2): + ms=[set(range(L))]*hubs; w=L//npieces + for p in range(npieces): ms+=piece(p*w,(p+1)*w,w//2) + return ms +def check(ms,L,verbose=True): + Gs=samples(L); viol=[] + for lab in itertools.product(range(4),repeat=L): + ma=sum(1<mb and False: continue + A=[i for i in range(L) if ma>>i&1];B=[i for i in range(L) if mb>>i&1] + need=minrank(Gs,A,B); have=sum(1 for m in ms if (m&set(A)) and (m&set(B))) + if have=c), -1 (a for all stage sets A,B: +#{r : M_r meets A and B} >= min_Q rank G~[A,B] (min over a sample of Q: I,2I,-I,.5I, 6 randoms). +Min header sum|Z_k| over mask multisets (ILP, HiGHS). Also optional layout check.""" +import sys, itertools, numpy as np +from scipy.optimize import milp, LinearConstraint, Bounds +rng=np.random.default_rng(1) +def Gt(L,Qs): + G=np.zeros((2*L,2*L)) + for a in range(L): + for c in range(L): + G[2*a:2*a+2,2*c:2*c+2]=(1 if a>=c else -1)*np.eye(2) + G[2*a:2*a+2,2*a:2*a+2]-=Qs[a] + return G +def samples(L): + S=[] + for s in (1,2,-1,.5): S.append([s*np.eye(2)]*L) + for _ in range(6): S.append([rng.normal(size=(2,2)) for _ in range(L)]) + for _ in range(4): S.append([np.array([[1,rng.normal()],[0,1]])*rng.choice([1,2,-1]) for _ in range(L)]) + return [Gt(L,q) for q in S] +def minrank(Gs,A,B): + idx=lambda X:[2*a+j for a in X for j in (0,1)] + return min(np.linalg.matrix_rank(G[np.ix_(idx(A),idx(B))],tol=1e-9) for G in Gs) +if __name__=='__main__': + L=int(sys.argv[1]); Gs=samples(L); masks=list(range(1,1<>i&1],[i for i in range(L) if mb>>i&1]) + if k==0: continue + rows.append([1.0 if (m&ma and m&mb) else 0.0 for m in masks]); rhs.append(k) + A_=np.array(rows); c=np.array([pc(m) for m in masks],float) + res=milp(c,constraints=LinearConstraint(A_,np.array(rhs),np.inf),integrality=np.ones(len(masks)),bounds=Bounds(0,60),options={'time_limit':900}) + print('L',L,'status',res.status,'min header',res.fun); x=np.round(res.x).astype(int) + print('Z',[int(sum(x[i] for i,m in enumerate(masks) if m>>k&1)) for k in range(L)],'n',x.sum()) + for i,m in enumerate(masks): + if x[i]: print(x[i],'x',''.join('1' if m>>k&1 else '.' for k in range(L))) diff --git a/research/p104-findings-nov15/code/dyingpair/lay.py b/research/p104-findings-nov15/code/dyingpair/lay.py new file mode 100644 index 0000000000..92e09b8542 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/lay.py @@ -0,0 +1,26 @@ +import sys,time,itertools +import numpy as np +from rs2 import * +from chk import check +# time order: stages 0..L-1 ; left arm = 0..nl-1 ; thin between nl-1 and nl ; right arm nl..L-1 +def build(rolesets,L,nl): + n=len(rolesets); Z=[[r for r,s in enumerate(rolesets) if k in s] for k in range(L)]; th=n+2-2*L + o=[('S',Z[k]) for k in range(nl)]+[('T',)]*th+[('S',Z[k]) for k in range(nl,L)] + return n,o,th +def B(L,nl): + ms=[set(range(L))]*2 + for k in range(nl): ms+=[set(range(k,nl))]*2 + for k in range(L-1,nl-1,-1): ms+=[set(range(nl,k+1))]*2 + return ms +def win(L,nl,w): + ms=[set(range(L))]*2 + for k in range(nl): ms+=[set(range(k,min(k+w,nl)))]*2 + for k in range(L-1,nl-1,-1): ms+=[set(range(max(k-w+1,nl),k+1))]*2 + return ms +if __name__=='__main__': + name=sys.argv[1]; starts=int(sys.argv[2]); L=int(sys.argv[3]); nl=L//2 + ms={'B':B(L,nl),'w1':win(L,nl,1),'w2':win(L,nl,2)}[name] + v=check(ms,L); print(name,'L',L,'counting violations',len(v),flush=True) + n,o,th=build(ms,L,nl); m=Model(n,o,-0.5,()); t0=time.time() + h,b,z=run(m,starts,seed=3,nfev=1500) + print('%s L=%d n=%d N=%d dim=%d solved %d/%d best %.1e (%.0fs)'%(name,L,n,m.N,m.dim,h,starts,b,time.time()-t0),flush=True) diff --git a/research/p104-findings-nov15/code/dyingpair/layout_price.py b/research/p104-findings-nov15/code/dyingpair/layout_price.py new file mode 100644 index 0000000000..3fb9a03d64 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/layout_price.py @@ -0,0 +1,88 @@ +"""CALLEROPEN layout_price.py [C-param, PRICEKIT]: price a rho=2 mask layout from (i) role masks (stage sets), (ii) per-boundary +line caps. Ledger [P]: T = sum_roles T[8 deg+1] (odd phase), idle incidences at stage k = |Z_k| - 2 (Lemma Det), thin = n+2-2L +QUERY P[1]; lines = intervals of consecutive stages, #lines at k = |Z_k|-2, #crossing k|k+1 <= cap_k. Line lengths: at each +boundary continue the OLDEST lines up to min(m_k, m_k+1, cap_k) (most unequal arrangement for convex f) -> P[8 len]. +Natives: FABLE5 allb_pk set (twins deleted / shadows / shadows+2 relays) for 104 ports, 17 calls.""" +import sys, copy, math +import os; _HERE = os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0, os.path.join(_HERE, '..')); import pricekit as pk +from collections import Counter +W = pk.load(os.path.join(_HERE, '..', '..', 'data', 'w7546.json.gz')); C7 = pk.load(os.path.join(_HERE, '..', '..', 'data', '7113_CALLER.json.gz')) +n0 = pk.native_from_word(W); base = pk.price(pk.invoice(pk.caller_invoice(C7), n0), 30); K = base['kappa']['lo'] +W2 = copy.deepcopy(W); W2['hosted'] = {}; nt = pk.native_from_word(W2) +nm = dict(nt); P_ = Counter(n0['P']) +for c in ('X2', 'Y2'): P_.subtract(n0['main_moves'][c]) +nm['P'] = +P_; nm['T'] = Counter(n0['T']); nm['source'] = 'param' +nr = dict(nt); nr['T'] = Counter(nt['T']); nr['P'] = Counter(nt['P']); nr['source'] = 'param' +nr['T'][9] += 2 * 96; nr['P'][1] += 96; nr['P'][8] += 2 * 96; nr['P'][1] += 96 +NAT = (('twins deleted', nm), ('shadows', nt), ('shadows+2 relays', nr)) +f = lambda r: r * math.log(r) if r > 1 else 0.0 +def lines(m, cap): + """m[k] lines needed at stage k (k=0..L-1); cap[k] max crossing k|k+1. Returns Counter of line lengths.""" + alive = [] # start stages, oldest first + out = Counter() + for k in range(len(m)): + if k > 0: + keep = min(m[k - 1], m[k], cap[k - 1], len(alive)) + for s in alive[keep:]: out[k - s] += 1 + alive = alive[:keep] + alive += [k] * (m[k] - len(alive)) + for s in alive: out[len(m) - s] += 1 + return out +def invoice(masks, L, cap, cov=2): + """masks: list of role stage-sets; cap: list of L-1 caps.""" + Z = [sum(1 for ms in masks if k in ms) for k in range(L)] + m = [z - cov for z in Z]; assert min(m) >= 0, Z + T = Counter(8 * len(ms) + 1 for ms in masks) + P = Counter({8 * l: v for l, v in lines(m, cap).items()}) + n = len(masks); thin = n + 2 - 2 * L; assert thin >= 0, (n, L) + # thin QUERYs absorbed as IDLE_ODD into the shortest lines (as the built B does: 4 x r9) + for _ in range(thin): + r = min((r for r in P if r % 8 == 0 and P[r] > 0), default=None) + if r is None: P[1] += 1; continue + P[r] -= 1; P[r + 1] += 1 + return T, +P, Z +def price(T, P, lab, nJ=17, quiet=False): + Ta = Counter({r: v * 104 for r, v in T.items()}); Pa = Counter({r: v * 104 for r, v in P.items()}) + res = [] + for nn, nat in NAT: + pr = pk.price(pk.invoice(dict(T=Ta, P=Pa, nJ=nJ), nat), 30); k = pr['kappa'] + res.append(float(k['lo'] / K) if isinstance(k, dict) and k.get('lo') else 0.0) + D = sum(v * f(r) for r, v in T.items()) - sum(v * f(r) for r, v in P.items()) + if not quiet: + print('%-58s S/port %4d D0/port %7.1f header %3d | td %.4f sh %.4f rel %.4f' % (lab, pk.S_of(T, P), D, sum((r - 1) // 8 * v for r, v in T.items()), *res)) + return res +def two_arm(L=17, nl=8, hubs=2, grow=2, extra=0, deg1=True): + """B-type: hubs on all stages; left arm stages 0..nl-1 (centre nl-1), right nl..L-1 (centre nl). At left stage k (0-based) + 'grow' own roles enter (k>=1 or always), mask k..nl-1; 'extra' own roles enter at stage 0 additionally.""" + masks = [set(range(L)) for _ in range(hubs)] + for k in range(nl): + cnt = grow + (extra if k == 0 else 0) + masks += [set(range(k, nl)) for _ in range(cnt)] + for k in range(L - 1, nl - 1, -1): + cnt = grow + (extra if k == L - 1 else 0) + masks += [set(range(nl, k + 1)) for _ in range(cnt)] + return masks +def caps_B(Z, L, nl, bonus=0): + """interior caps |S|-4 ; centre boundary: S = hubs only -> crossing <= hubs-? use |S|-4 floored at 0; outer boundaries |S|-4+bonus.""" + cap = [] + for k in range(L - 1): + S = Z[k] if k < nl - 1 else (Z[k + 1] if k >= nl else None) + if S is None: cap.append(0); continue # centre (thin block between): no line crosses (S = hubs, covered) + c = S - 4 + if k == 0 or k == L - 2: c += bonus + cap.append(max(c, 0)) + return cap +if __name__ == '__main__': + print(pk.cite('control #7546 on #7113', base)) + Zof = lambda ms: [sum(1 for m in ms if k in m) for k in range(17)] + ms = two_arm(); T, P, Z = invoice(ms, 17, caps_B(Zof(ms), 17, 8)) + print('model B: Z', Z, 'P', sorted(P.items())) + price(T, P, 'model B (built B: 1.0950/1.0031/0.9773)') + T, P, Z = invoice(ms, 17, caps_B(Zof(ms), 17, 8, bonus=2)); price(T, P, 'B + outer bonus (2 lines cross each outer boundary)') + T, P, Z = invoice(ms, 17, caps_B(Zof(ms), 17, 8, bonus=1)); price(T, P, 'B + outer bonus 1') + ms2 = two_arm(extra=-2); Z2 = Zof(ms2); print('tight Z', Z2, 'n', len(ms2)) + T, P, Z = invoice(ms2, 17, caps_B(Z2, 17, 8)); price(T, P, 'tight two-hub |Z_k|=2k (no own role at outer stages)') + for nl in (7, 9): + ms3 = two_arm(nl=nl); T, P, Z = invoice(ms3, 17, caps_B(Zof(ms3), 17, nl)); price(T, P, 'B with centre at %d|%d' % (nl, nl + 1)) + # uncapped ceiling (all lines run full) for reference + T, P, Z = invoice(ms, 17, [99] * 16); price(T, P, 'B masks, NO caps (unreachable reference)') diff --git a/research/p104-findings-nov15/code/dyingpair/nn2.py b/research/p104-findings-nov15/code/dyingpair/nn2.py new file mode 100644 index 0000000000..84f8269962 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/nn2.py @@ -0,0 +1,16 @@ +import sys,time +import numpy as np +from rs2 import * +from lay import build,B +def s(x): return {i for i,c in enumerate(x) if c=='1'} +LAY={'ilpmin':['1...','.1..','..1.','.11.','...1','1..1','1111','1111'], + 'srch':['1...','1...','.1..','..1.','111.','...1','...1','1..1','.1.1','..11','1111']} +if __name__=='__main__': + name=sys.argv[1]; comm=int(sys.argv[2]); pos=int(sys.argv[3]); starts=int(sys.argv[4]); L=4 + ms=B(4,2) if name=='B' else [s(x) for x in LAY[name]] + n=len(ms); Z=[[r for r,t in enumerate(ms) if k in t] for k in range(L)]; th=n+2-2*L + o=[('S',Z[k]) for k in range(pos)]+[('T',)]*th+[('S',Z[k]) for k in range(pos,L)] + m=Model(n,o,-0.5,()); m.comm=bool(comm); t0=time.time() + h,b,z=run(m,starts,seed=5,nfev=1500) + print('%s comm=%d pos=%d n=%d N=%d dim=%d solved %d/%d best %.1e (%.0fs)'%(name,comm,pos,n,m.N,m.dim,h,starts,b,time.time()-t0),flush=True) + if h: np.save('sol_%s_%d_%d.npy'%(name,comm,pos),z) diff --git a/research/p104-findings-nov15/code/dyingpair/nn_rs.py b/research/p104-findings-nov15/code/dyingpair/nn_rs.py new file mode 100644 index 0000000000..10b1cbf009 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/nn_rs.py @@ -0,0 +1,21 @@ +"""NONNEST: exact-ish (float 1e-10) feasibility rows for layouts given as role stage-sets. Thin block position pos in 0..L.""" +import sys, time, json +from rs2 import * +def build(rolesets, L, pos): + n=len(rolesets); Z=[[r for r,s in enumerate(rolesets) if k in s] for k in range(L)]; th=n+2-2*L + o=[('S',Z[k]) for k in range(pos)]+[('T',)]*th+[('S',Z[k]) for k in range(pos,L)] + return n,o +def s(x): return {i for i,c in enumerate(x) if c=='1'} +LAY={ + 'ctlB4': None, + 'ilpmin': ['1...','.1..','..1.','.11.','...1','1..1','1111','1111'], + 'srch': ['1...','.1..','..1.','.1..','...1','...1','111.','1..1','1111','.1.1','..11'][:0] or + ['1...','1...','.1..','..1.','111.','...1','...1','1..1','.1.1','..11','1111'], +} +if __name__=='__main__': + name=sys.argv[1]; pos=int(sys.argv[2]); starts=int(sys.argv[3]); L=4 + if name=='ctlB4': n,o,_=two_arm(4,2) + else: n,o=build([s(x) for x in LAY[name]],L,pos) + m=Model(n,o,-0.5,()); t0=time.time(); h,b,z=run(m,starts,seed=5,nfev=1500) + print('%s pos=%d n=%d N=%d vars=%d solved %d/%d best %.1e (%.0fs)'%(name,pos,n,m.N,m.dim,h,starts,b,time.time()-t0),flush=True) + if h: np.save('sol_%s_%d.npy'%(name,pos),z) diff --git a/research/p104-findings-nov15/code/dyingpair/o_s2_8_4.txt b/research/p104-findings-nov15/code/dyingpair/o_s2_8_4.txt new file mode 100644 index 0000000000..6925bd734e --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/o_s2_8_4.txt @@ -0,0 +1,13 @@ +B control L=8 nl=4 violations 0 +c=1 k=0 d=0 arms=1 violations 0 [] (12s) +c=1 k=0 d=0 arms=2 violations 0 [] (12s) +c=1 k=0 d=1 arms=1 violations 0 [] (12s) +c=1 k=0 d=1 arms=2 violations 0 [] (12s) +c=1 k=1 d=1 arms=1 violations 0 [] (12s) +c=1 k=1 d=1 arms=2 violations 0 [] (12s) +c=2 k=0 d=0 arms=1 violations 0 [] (12s) +c=2 k=0 d=0 arms=2 violations 170 [([0, 6], [3, 7], np.int64(4), 2), ([0, 5], [3, 7], np.int64(4), 2)] (12s) +c=2 k=0 d=1 arms=1 violations 0 [] (12s) +c=2 k=0 d=1 arms=2 violations 18 [([0, 5], [3, 7], np.int64(4), 2), ([0, 4], [3, 7], np.int64(4), 2)] (12s) +c=2 k=1 d=1 arms=1 violations 0 [] (12s) +c=2 k=1 d=1 arms=2 violations 9 [([0, 1, 5, 7], [0, 3, 6, 7], np.int64(8), 6), ([0, 1, 4, 7], [0, 3, 6, 7], np.int64(8), 6)] (12s) diff --git a/research/p104-findings-nov15/code/dyingpair/o_s2_8_5.txt b/research/p104-findings-nov15/code/dyingpair/o_s2_8_5.txt new file mode 100644 index 0000000000..e2d62c7964 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/o_s2_8_5.txt @@ -0,0 +1,15 @@ +B control L=8 nl=5 violations 0 +c=1 k=0 d=0 arms=1 violations 0 [] (14s) +c=1 k=0 d=0 arms=2 violations 0 [] (12s) +c=1 k=0 d=1 arms=1 violations 0 [] (12s) +c=1 k=0 d=2 arms=1 violations 0 [] (12s) +c=1 k=1 d=1 arms=1 violations 0 [] (13s) +c=1 k=1 d=2 arms=1 violations 0 [] (12s) +c=1 k=2 d=2 arms=1 violations 0 [] (12s) +c=2 k=0 d=0 arms=1 violations 0 [] (12s) +c=2 k=0 d=0 arms=2 violations 149 [([0, 6], [4, 7], np.int64(4), 2), ([0, 5], [4, 7], np.int64(4), 2)] (12s) +c=2 k=0 d=1 arms=1 violations 0 [] (12s) +c=2 k=0 d=2 arms=1 violations 0 [] (12s) +c=2 k=1 d=1 arms=1 violations 0 [] (12s) +c=2 k=1 d=2 arms=1 violations 0 [] (12s) +c=2 k=2 d=2 arms=1 violations 0 [] (12s) diff --git a/research/p104-findings-nov15/code/dyingpair/o_s2_9_5.txt b/research/p104-findings-nov15/code/dyingpair/o_s2_9_5.txt new file mode 100644 index 0000000000..d90a133211 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/o_s2_9_5.txt @@ -0,0 +1,10 @@ +B control L=9 nl=5 violations 0 +c=1 k=0 d=0 arms=1 violations 0 [] (52s) +c=1 k=0 d=0 arms=2 violations 0 [] (52s) +c=1 k=0 d=1 arms=1 violations 0 [] (52s) +c=1 k=0 d=1 arms=2 violations 0 [] (52s) +c=1 k=0 d=2 arms=1 violations 0 [] (53s) +c=1 k=1 d=1 arms=1 violations 0 [] (53s) +c=1 k=1 d=1 arms=2 violations 0 [] (53s) +c=1 k=1 d=2 arms=1 violations 0 [] (53s) +c=1 k=2 d=2 arms=1 violations 0 [] (53s) diff --git a/research/p104-findings-nov15/code/dyingpair/o_s2_L6.txt b/research/p104-findings-nov15/code/dyingpair/o_s2_L6.txt new file mode 100644 index 0000000000..999323a63c --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/o_s2_L6.txt @@ -0,0 +1,7 @@ +B control L=6 nl=4 violations 0 +c=1 k=0 d=0 arms=1 violations 0 [] (1s) +c=1 k=0 d=1 arms=1 violations 0 [] (1s) +c=1 k=1 d=1 arms=1 violations 0 [] (1s) +c=2 k=0 d=0 arms=1 violations 0 [] (1s) +c=2 k=0 d=1 arms=1 violations 0 [] (1s) +c=2 k=1 d=1 arms=1 violations 0 [] (1s) diff --git a/research/p104-findings-nov15/code/dyingpair/o_s2_L7.txt b/research/p104-findings-nov15/code/dyingpair/o_s2_L7.txt new file mode 100644 index 0000000000..ce3a97664f --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/o_s2_L7.txt @@ -0,0 +1,7 @@ +B control L=7 nl=4 violations 0 +c=1 k=0 d=0 arms=1 violations 0 [] (3s) +c=1 k=0 d=1 arms=1 violations 0 [] (3s) +c=1 k=1 d=1 arms=1 violations 0 [] (3s) +c=2 k=0 d=0 arms=1 violations 0 [] (3s) +c=2 k=0 d=1 arms=1 violations 0 [] (3s) +c=2 k=1 d=1 arms=1 violations 0 [] (3s) diff --git a/research/p104-findings-nov15/code/dyingpair/o_s3_B.txt b/research/p104-findings-nov15/code/dyingpair/o_s3_B.txt new file mode 100644 index 0000000000..d0e94a2525 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/o_s3_B.txt @@ -0,0 +1,25 @@ +B Z [4, 6, 8, 10, 6, 4] n 14 thin 4 dim 264 +start 0 solved 0 best 9.7e-01 (28s) +start 1 solved 0 best 1.8e+00 (30s) +start 2 solved 0 best 1.7e+00 (29s) +start 3 solved 0 best 1.5e-02 (31s) +start 4 solved 0 best 8.7e-01 (30s) +start 5 solved 0 best 1.4e-02 (29s) +start 6 solved 0 best 9.6e-01 (30s) +start 7 solved 0 best 1.0e+00 (29s) +start 8 solved 0 best 1.5e+00 (30s) +start 9 solved 0 best 8.3e-01 (30s) +start 10 solved 0 best 1.1e+00 (31s) +start 11 solved 0 best 1.4e+00 (30s) +start 12 solved 0 best 1.3e-02 (31s) +start 13 solved 0 best 1.2e-01 (28s) +start 14 solved 0 best 1.8e+00 (32s) +start 15 solved 0 best 2.2e-01 (33s) +start 16 solved 0 best 7.1e-01 (34s) +start 17 solved 0 best 8.6e-01 (31s) +start 18 solved 0 best 7.1e-01 (31s) +start 19 solved 0 best 1.1e+00 (30s) +start 20 solved 0 best 8.6e-01 (31s) +start 21 solved 0 best 8.6e-01 (34s) +start 22 solved 0 best 2.2e-02 (29s) +start 23 solved 0 best 9.7e-01 (32s) diff --git a/research/p104-findings-nov15/code/dyingpair/o_s3_probe.txt b/research/p104-findings-nov15/code/dyingpair/o_s3_probe.txt new file mode 100644 index 0000000000..154975ad4c --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/o_s3_probe.txt @@ -0,0 +1,20 @@ +B Z [4, 6, 8, 10, 6, 4] n 14 thin 4 dim 264 +start 0 solved 0 best 9.7e-01 (28s) + 28.57 real 26.93 user 1.61 sys + 174555136 maximum resident set size + 0 average shared memory size + 0 average unshared data size + 0 average unshared stack size + 15978 page reclaims + 99 page faults + 0 swaps + 0 block input operations + 0 block output operations + 0 messages sent + 0 messages received + 0 signals received + 199 voluntary context switches + 108 involuntary context switches + 372753271403 instructions retired + 94506632729 cycles elapsed + 74876160 peak memory footprint diff --git a/research/p104-findings-nov15/code/dyingpair/price_nn.py b/research/p104-findings-nov15/code/dyingpair/price_nn.py new file mode 100644 index 0000000000..adbca7f3f3 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/price_nn.py @@ -0,0 +1,24 @@ +import sys; sys.path.insert(0,'.') +import layout_price as lp +from layout_price import * +L=17; nl=8 +Zof=lambda ms:[sum(1 for m in ms if k in m) for k in range(L)] +def win(w, grow=2, hubs=2): + """sliding-window arms: role introduced at left stage k lives on stages k..min(k+w-1,nl-1); right mirrored.""" + ms=[set(range(L)) for _ in range(hubs)] + for k in range(nl): ms+=[set(range(k,min(k+w,nl)))]*grow + for k in range(L-1,nl-1,-1): ms+=[set(range(max(k-w+1,nl),k+1))]*grow + return ms +def capS(ms, mode): + Z=Zof(ms); cap=[] + for k in range(L-1): + if k==nl-1: cap.append(0); continue + S=sum(1 for m in ms if k in m and k+1 in m) + cap.append(max(S-4,0) if mode=='lemma' else (max(S-2,0) if mode=='opt' else 99)) + return cap +ms=two_arm(); T,P,Z=invoice(ms,L,caps_B(Zof(ms),L,nl)); price(T,P,'B control') +for w in (1,2,3,4,8): + ms=win(w) + for mode in ('lemma','opt','none'): + try: T,P,Z=invoice(ms,L,capS(ms,mode)); price(T,P,'window w=%d caps=%s Zmax %d'%(w,mode,max(Z))) + except AssertionError as e: print('w',w,mode,'assert',e) diff --git a/research/p104-findings-nov15/code/dyingpair/qcheck.py b/research/p104-findings-nov15/code/dyingpair/qcheck.py new file mode 100644 index 0000000000..99ab35a1db --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/qcheck.py @@ -0,0 +1,15 @@ +"""[C] check proof ingredients on the nested B L=4 solution (NONNEST sol_ctlB4_2.npy, t=-0.5): K=I+L-U-diag rank 2, Q_a eigenvalues 1+-sqrt(t/(1+t)) (never 2), +Floor independence of X_0..X_k, X_r|H for k=0 (left arm 0..1, r = right stage).""" +import sys; sys.path.insert(0,'.'); import numpy as np +from rs2 import Model, two_arm +n,o,th=two_arm(4,2); m=Model(n,o,-0.5,()); z=np.load('sol_ctlB4_2.npy'); X,Y=m.unpack(z) +print('residual',np.max(np.abs(m.fj(z)[0]))) +G=Y@X.T; N=m.N; K=np.eye(N)+np.tril(G,-1)-np.triu(G,1)-np.diag(np.diag(G)) +U,s,Vt=np.linalg.svd(K); print('rank K sv', np.round(s[:4],6)); W=U[:,:2]*s[:2]; Zm=Vt[:2].T +for st,(u0,mask) in enumerate(m.stages): + i=[u0,u0+1]; Gs=G[np.ix_(i,i)]; Wa=W[i]; Za=Zm[i]; Q=np.eye(2)-np.linalg.inv(Wa)@Gs@np.linalg.inv(Za.T) + t=Gs[0,1]*Gs[1,0]; print('stage',st,'t',round(t,6),'eig Q',np.round(np.linalg.eigvals(Q),6),'pred',np.round([1+np.sqrt(complex(t/(1+t))),1-np.sqrt(complex(t/(1+t)))],6)) +# Floor k=0: columns X_0 (2), X_r|H (2) with rows Y_0,Y_1 ; H = hubs {0,1} +st=m.stages; r=st[-1][0]; H=[0,1] +cols=[X[st[0][0]],X[st[0][0]+1]]+[np.where(np.isin(np.arange(n),H),X[u],0) for u in (r,r+1)] +print('Floor k=0 rank', np.linalg.matrix_rank(np.array(cols),tol=1e-8),'of 4') diff --git a/research/p104-findings-nov15/code/dyingpair/rs2.py b/research/p104-findings-nov15/code/dyingpair/rs2.py new file mode 100644 index 0000000000..eb38707006 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/rs2.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +"""CALLEROPEN rs2.py [N]: same model as rootsearch.py (units, product -I, HOST pair invariant, supports in masks, PREFIX +COMMUTATION before/after every stage, optional crossing idle lines) with an ANALYTIC Jacobian. Solver scipy least_squares +(trf), fixed number of random starts per row (stopping rule). solved = max|residual| < 1e-10.""" +import sys, json +import numpy as np +from scipy.optimize import least_squares +class Model: + def __init__(s, n, order, t, lines=()): + s.n = n; units = []; stages = [] + for it in order: + if it[0] == 'S': stages.append((len(units), list(it[1]))); units += [list(it[1]), list(it[1])] + else: units.append(list(range(n)) if len(it) == 1 else list(it[1])) + s.units = units; s.stages = stages; s.N = N = len(units) + s.idx = []; off = 0 + for sup in units: s.idx.append((off, np.array(sup))); off += 2 * len(sup) + s.lidx = [] + for (a, b, c) in lines: + S = sorted(set(stages[a][1]) & set(stages[b][1])) + for _ in range(c): s.lidx.append((a, b, off, np.array(S))); off += 2 * len(S) + s.dim = off; s.tk = t if isinstance(t, (list, tuple)) else [t] * len(stages) + def unpack(s, z): + X = np.zeros((s.N, s.n)); Y = np.zeros((s.N, s.n)) + for i, (o, sup) in enumerate(s.idx): + k = len(sup); X[i, sup] = z[o:o + k]; Y[i, sup] = z[o + k:o + 2 * k] + return X, Y + def fj(s, z): + n, N = s.n, s.N; X, Y = s.unpack(z); I = np.eye(n) + T = [I - 2 * np.outer(X[i], Y[i]) for i in range(N)] + pref = [I] + for i in range(N): pref.append(T[i] @ pref[-1]) + rows = []; jac = [] + def blank(m): return np.zeros((m, s.dim)) + # 1 isotropy + r = np.sum(X * Y, axis=1); J = blank(N) + for i, (o, sup) in enumerate(s.idx): + k = len(sup); J[i, o:o + k] = Y[i, sup]; J[i, o + k:o + 2 * k] = X[i, sup] + rows.append(r); jac.append(J) + # product and prefixes: d(A_{u}) where A_u = T_{u-1}..T_0 ; dA_u/dunit i (i 1 else 6 +Ls = [int(a) for a in sys.argv[2].split(',')] if len(sys.argv) > 2 else [4, 5, 6] +out = [] +for L in Ls: + nl = L // 2; rows = [] + n, o, th = two_arm(L, 2); Z = [it[1] for it in o if it[0] == 'S'] + rows.append(('ctl B two-hub', n, o, ())) + rows.append(('neg one-hub (Thm 1\')', *two_arm(L, 1)[:2], ())) + rows.append(('TEST tight (outer masks = hubs only)', *two_arm(L, 2, outer_own=0)[:2], ())) + rows.append(('TEST B + 2 lines across LEFT outer bdry', n, o, ((0, 1, 2),))) + rows.append(('TEST B + 1 line across LEFT outer bdry', n, o, ((0, 1, 1),))) + rows.append(('TEST B + 2 lines across RIGHT outer bdry', n, o, ((L - 2, L - 1, 2),))) + if nl >= 3: + S = len(set(Z[1]) & set(Z[2])) + rows.append(('ctl B, |S|-4=%d lines across left 1|2' % (S - 4), n, o, ((1, 2, S - 4),))) + rows.append(('TEST B, |S|-3=%d lines across left 1|2 (Lemma C: no)' % (S - 3), n, o, ((1, 2, S - 3),))) + for lab, n_, o_, ln in rows: + t0 = time.time(); m = Model(n_, o_, -0.5, ln); h, b, z = run(m, STARTS) + print('L=%d %-48s n=%2d N=%2d vars %4d solved %d/%d best %.1e (%.0fs)' % (L, lab, n_, m.N, m.dim, h, STARTS, b, time.time() - t0), flush=True) + out.append(dict(L=L, row=lab, n=n_, N=m.N, vars=m.dim, solved=h, starts=STARTS, best=b)) + if h: np.save('sol_L%d_%s.npy' % (L, ''.join(c for c in lab if c.isalnum())[:30]), z) +json.dump(out, open('rs2_%s.json' % '_'.join(map(str, Ls)), 'w'), indent=1) diff --git a/research/p104-findings-nov15/code/dyingpair/sol_ctlB4_2.npy b/research/p104-findings-nov15/code/dyingpair/sol_ctlB4_2.npy new file mode 100644 index 0000000000..838cd934d7 Binary files /dev/null and b/research/p104-findings-nov15/code/dyingpair/sol_ctlB4_2.npy differ diff --git a/research/p104-findings-nov15/code/dyingpair/step1.py b/research/p104-findings-nov15/code/dyingpair/step1.py new file mode 100644 index 0000000000..7a7edd6efe --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/step1.py @@ -0,0 +1,44 @@ +"""DYINGPAIR step 1 [C-param] (asserts the B control and the 35-of-63 pass count; prints PASS_DYINGPAIR_PRICE): price two-arm B layouts (L=17, nl=8) where c own roles entering left stage k die after stage d (< nl-1). +Caps: 'lemma' |S_k|-4 with S_k = roles alive on both sides of the boundary (NONNEST capS). Penalty variant: DYINGGEN Lemma 2 adds |E|-1 +extra roles (here 1 for a pair, entering at d+1 and living to the centre); Floor' check |Z_j| >= 2(j+1)+2-b_j with b_j <= #dead roles.""" +import sys, os; sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from layout_price import invoice, price, two_arm +L=17; nl=8 +Zof=lambda ms:[sum(1 for m in ms if k in m) for k in range(L)] +def capS(ms): + cap=[] + for k in range(L-1): + if k==nl-1: cap.append(0); continue + S=sum(1 for m in ms if k in m and k+1 in m); cap.append(max(S-4,0)) + return cap +def dying(specs): + """specs: list of (arm, k, d, c, extra): in arm 'L' (stages 0..nl-1) or 'R' (mirror), c of the roles entering at arm-depth k now live k..d; extra roles enter at d+1.""" + ms=two_arm() + for arm,k,d,c,extra in specs: + if arm=='L': full=set(range(k,nl)); new=set(range(k,d+1)); ex=set(range(d+1,nl)) + else: full=set(range(nl,L-k)); new=set(range(L-1-d,L-k)); ex=set(range(nl,L-1-d)) + for _ in range(c): ms.remove(full); ms.append(set(new)) + for _ in range(extra): ms.append(set(ex)) + return ms +def floor_ok(ms, ndead): + Z=Zof(ms); return all(Z[j]>=2*(j+1)+2-ndead for j in range(nl)) +def go(lab, specs, ndead): + ms=dying(specs); T,P,Z=invoice(ms,L,capS(ms)); r=price(T,P,lab+(' FLOOR-FAIL' if not floor_ok(ms,ndead) else ''),quiet=False); return r +B=go('B control', [], 0) +best=[] +for c in (1,2): + for k in range(0,nl-2): + for d in range(k,nl-2): + for extra in ((0,1) if c==2 else (0,)): + r=go('%s k=%d d=%d extra=%d (one arm)'%('pair' if c==2 else 'single',k,d,extra),[('L',k,d,c,extra)],c) + best.append((min(x/y for x,y in zip(r,B)),c,k,d,extra)) +print('--- per-layout min ratio over 3 columns vs B, >= 1.005 passes:') +for t in sorted(best,reverse=True)[:12]: print(' %.4f c=%d k=%d d=%d extra=%d'%t) +print('pass count', sum(1 for t in best if t[0]>=1.005), 'of', len(best)) +for (k,d) in ((0,3),(1,4),(0,1),(1,2)): + for extra in (0,1): + r=go('pair per arm k=%d d=%d extra=%d'%(k,d,extra),[('L',k,d,2,extra),('R',k,d,2,extra)],2) + +assert ['%.4f' % x for x in B] == ['1.0950', '1.0031', '0.9773'] +assert sum(1 for t in best if t[0] >= 1.005) == 35 and len(best) == 63 +print('PASS_DYINGPAIR_PRICE') diff --git a/research/p104-findings-nov15/code/dyingpair/step1b.py b/research/p104-findings-nov15/code/dyingpair/step1b.py new file mode 100644 index 0000000000..345b966614 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/step1b.py @@ -0,0 +1,15 @@ +"""escape pricing: dying pair/single entering k, dying after d, plus ONE slack role entering at stage j_e<=d (mask j_e..e_end).""" +import sys; sys.path.insert(0,'.') +from step1 import dying, go, L, nl +import step1 +B=go('B control',[],0) +for c in (2,1): + for k in (0,1): + for d in range(max(k,1),nl-2): + for je in range(k+1 if k==0 else 0, d+1): + for eend in (d, nl-1): + sp=[('L',k,d,c,0)]; ms=dying(sp); ms.append(set(range(je,eend+1))) + from layout_price import invoice, price + T,P,Z=invoice(ms,L,step1.capS(ms)); r=price(T,P,'c=%d k=%d d=%d slack@%d..%d'%(c,k,d,je,eend),quiet=True) + m=min(x/y for x,y in zip(r,B)) + if m>=1.005: print('PASS c=%d k=%d d=%d slack %d..%d min ratio %.4f td %.4f'%(c,k,d,je,eend,m,r[0]/B[0])) diff --git a/research/p104-findings-nov15/code/dyingpair/step2.py b/research/p104-findings-nov15/code/dyingpair/step2.py new file mode 100644 index 0000000000..638ad252f1 --- /dev/null +++ b/research/p104-findings-nov15/code/dyingpair/step2.py @@ -0,0 +1,27 @@ +"""DYINGPAIR step 2 [C chk.check, Block Rank Bound, 4^L]: unbalanced L<=9 analogues of B with c own roles entering left stage k dying after d.""" +import sys, time; sys.path.insert(0,'.') +from chk import check +def B(L,nl): + ms=[set(range(L))]*2 + for k in range(nl): ms+=[set(range(k,nl))]*2 + for k in range(L-1,nl-1,-1): ms+=[set(range(nl,k+1))]*2 + return [set(m) for m in ms] +def dying(L,nl,specs): + ms=B(L,nl) + for arm,k,d,c,extra in specs: + if arm=='L': full=set(range(k,nl)); new=set(range(k,d+1)); ex=set(range(d+1,nl)) + else: full=set(range(nl,L-k)); new=set(range(L-1-d,L-k)); ex=set(range(nl,L-1-d)) + for _ in range(c): ms.remove(full); ms.append(set(new)) + for _ in range(extra): ms.append(set(ex)) + return ms +if __name__=='__main__': + L=int(sys.argv[1]); nl=int(sys.argv[2]) + print('B control L=%d nl=%d violations'%(L,nl), len(check(B(L,nl),L)), flush=True) + for c in (1,2): + for k in range(0,nl-2): + for d in range(k,nl-2): + for both in (0,1): + sp=[('L',k,d,c,0)]+([('R',k,d,c,0)] if both and d3 else 7; L=6; nl=4 +ms=[set(m) for m in B(L,nl)] +if name in ('dp','ds','dpx','dsx'): + c=2 if name.startswith('dp') else 1 + for _ in range(c): ms.remove(set(range(0,nl))); ms.append({0,1}) + if name.endswith('x'): ms.append({1}) +n,o,th=build(ms,L,nl); m=Model(n,o,-0.5,()) +print(name,'Z',[sum(1 for s in ms if k in s) for k in range(L)],'n',n,'thin',th,'dim',m.dim,flush=True) +for i in range(starts): + t0=time.time(); h,b,z=run(m,1,seed=seed*1000+i,nfev=1500) + print('start %d solved %d best %.1e (%.0fs)'%(i,h,b,time.time()-t0),flush=True) + if h: np.save('sol_%s_%d.npy'%(name,i),z) diff --git a/research/p104-findings-nov15/code/gaplemma/chk2.py b/research/p104-findings-nov15/code/gaplemma/chk2.py new file mode 100644 index 0000000000..d21c92a589 --- /dev/null +++ b/research/p104-findings-nov15/code/gaplemma/chk2.py @@ -0,0 +1,13 @@ +"""GAPLEMMA chk2.py: F1 hub checks + structure on loc2 solutions.""" +import numpy as np, glob, sys +sys.argv=['x']; +exec(open('loc2.py').read().split('rng=')[0]) +for f in sorted(glob.glob('loc2_sol*.npy')): + z=np.load(f); X,Y,W,Zm,b1,b2,u=unpack(z); G=Y.T@X + t=[G[2*s,2*s+1]*G[2*s+1,2*s] for s in range(3)] + H=np.zeros((n,2));H[0,0]=H[1,1]=1 + f1=[np.linalg.svd(np.hstack([X[:,:2*j+2],H]),compute_uv=False)[-1] for j in range(3)] + f1y=[np.linalg.svd(np.hstack([Y[:,:2*j+2],H]),compute_uv=False)[-1] for j in range(3)] + rXD=np.linalg.matrix_rank(X[[4,5,6]],1e-7); rYD=np.linalg.matrix_rank(Y[[4,5,6]],1e-7) + xs0=np.abs(X[4,:2]).max(); Ds=np.abs(X[[5,6]][:, :2]).max() + print(f,'t',np.round(t,3),'F1x min sv',np.round(f1,3),'F1y',np.round(f1y,3),'rank X[Delta]',rXD,'Y[Delta]',rYD,'|x_0s| %.2f'%xs0) diff --git a/research/p104-findings-nov15/code/gaplemma/chk3.py b/research/p104-findings-nov15/code/gaplemma/chk3.py new file mode 100644 index 0000000000..385feaa145 --- /dev/null +++ b/research/p104-findings-nov15/code/gaplemma/chk3.py @@ -0,0 +1,9 @@ +import numpy as np, sys +sys.argv=['x']; exec(open('loc2.py').read().split('rng=')[0]) +H=np.zeros((n,2));H[0,0]=H[1,1]=1 +for f in ['loc2_sol5.npy','loc2_sol8.npy','loc2_sol21.npy','loc2_sol27.npy']: + z=np.load(f); X,Y,W,Zm,b1,b2,u=unpack(z) + for nm,M,ZZ in (('X',X,Zm),('Y',Y,W)): + A=np.hstack([M,H]); U,S,Vt=np.linalg.svd(A); k=Vt[-1]; beta=k[:6] + alpha=[np.linalg.solve(ZZ[2*s:2*s+2].T, beta[2*s:2*s+2]) if False else ZZ[2*s:2*s+2].T@beta[2*s:2*s+2] for s in range(3)] + print(f,nm,'sv %.1e'%S[-1],'beta',np.round(beta,3),'Z^T beta (stage sums)',np.round(sum(alpha),4),'vec',np.round(M@beta,3)) diff --git a/research/p104-findings-nov15/code/gaplemma/chkA.py b/research/p104-findings-nov15/code/gaplemma/chkA.py new file mode 100644 index 0000000000..378f1d593f --- /dev/null +++ b/research/p104-findings-nov15/code/gaplemma/chkA.py @@ -0,0 +1,8 @@ +import numpy as np, sys, glob +spec=sys.argv[1]; sys.argv=['x',spec,'0','0'] +exec(open('locgen.py').read().split('rng=')[0]) +for f in sorted(glob.glob('loc_%s_*.npy'%spec)): + z=np.load(f); X,Y,W,Zm,B=unpack(z); G=Y.T@X + print(f,'max|res| %.1e'%np.abs(res(z)).max(),'t',[round(G[2*s,2*s+1]*G[2*s+1,2*s],3) for s in range(K)]) + print(' X rows Delta\n',np.round(X[Dl],4),'\n Y rows Delta\n',np.round(Y[Dl],4)) + print(' smin X',np.linalg.svd(X,compute_uv=False)[-1], 'B vec', np.round(X@B,4).T) diff --git a/research/p104-findings-nov15/code/gaplemma/codim.py b/research/p104-findings-nov15/code/gaplemma/codim.py new file mode 100644 index 0000000000..76353eb53b --- /dev/null +++ b/research/p104-findings-nov15/code/gaplemma/codim.py @@ -0,0 +1,20 @@ +"""GAPLEMMA codim.py [N]: at the family-A d=2 local root (loc_A2b2_7), dimension of the set of M = P_3|Z reachable by the left arm +with P_3 = diag(M, -I_Delta) (supports + x.y = 0 only), vs the 40-dim variety {M on R^7 : rank(M-I) = 4}.""" +import numpy as np, sys +sys.argv=['x','A2b2','0','0']; exec(open('locgen.py').read().split('rng=')[0]) +z=np.load('loc_A2b2_7.npy'); u=z[:nxy] +Zc=[i for i in range(n) if i not in Dl] +def F(u): + zz=z.copy(); zz[:nxy]=u; X,Y,W,Zm,B=unpack(zz); P=np.eye(n) + for i in range(NU): P=(np.eye(n)-2*np.outer(X[:,i],Y[:,i]))@P + con=np.concatenate([np.sum(X*Y,0),(P[np.ix_(Dl,Dl)]+np.eye(2)).ravel(),P[np.ix_(Dl,Zc)].ravel(),P[np.ix_(Zc,Dl)].ravel()]) + return con,P[np.ix_(Zc,Zc)].ravel() +def jac(f,u,h=1e-6): + c0,m0=f(u); Jc=np.zeros((len(c0),len(u))); Jm=np.zeros((len(m0),len(u))) + for i in range(len(u)): + e=np.zeros(len(u)); e[i]=h; c1,m1=f(u+e); c2,m2=f(u-e); Jc[:,i]=(c1-c2)/(2*h); Jm[:,i]=(m1-m2)/(2*h) + return Jc,Jm +Jc,Jm=jac(F,u); print('constraint residual',np.abs(F(u)[0]).max()) +sv=np.linalg.svd(Jc,compute_uv=False); rc=int((sv>1e-7*sv[0]).sum()); print('params',len(u),'constraint rank',rc) +_,_,Vt=np.linalg.svd(Jc); N=Vt[rc:].T +svm=np.linalg.svd(Jm@N,compute_uv=False); print('dim reachable M (tangent) =',int((svm>1e-7*svm[0]).sum()),' sv tail',np.round(svm[-6:],8)) diff --git a/research/p104-findings-nov15/code/gaplemma/dfloor.py b/research/p104-findings-nov15/code/gaplemma/dfloor.py new file mode 100644 index 0000000000..aaecdbbf58 --- /dev/null +++ b/research/p104-findings-nov15/code/gaplemma/dfloor.py @@ -0,0 +1,34 @@ +"""GAPLEMMA dfloor.py [P-criterion + C-param]: Delta-Floor kill (PROOF.md here) on the SLACK run6 menu. +Kill if exists 1<=i<=d: |Delta_{i-1}| < 2i+4 - |U_i minus Delta| (Delta = left roles of U_d absent from Z_K, K=d+1).""" +import sys, io, contextlib +import os; _SLACK = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'slack'); os.chdir(_SLACK); sys.path.insert(0, _SLACK) +with contextlib.redirect_stdout(io.StringIO()): + import run6 +from run6 import * +def dkill(ms,d): + K=d+1; lm=[m for m in left(ms)] + Dl=[m for m in lm if min(m)<=d and K not in m] + for i in range(0,d+1): + Ui=[m for m in ms if min(m)<=i and min(m)= +0.5%%: %d'%sum(1 for t in rows if t[0]>=0.5)) diff --git a/research/p104-findings-nov15/code/gaplemma/dfloor_count.py b/research/p104-findings-nov15/code/gaplemma/dfloor_count.py new file mode 100644 index 0000000000..f71e814377 --- /dev/null +++ b/research/p104-findings-nov15/code/gaplemma/dfloor_count.py @@ -0,0 +1,71 @@ +"""GAPLEMMA dfloor_count.py [C, stdlib, no pricing]: the purely combinatorial part of the kill chain on the pair/single + slack menu +of the L = 17, nl = 8 two-arm B layout (masks as sets of stages; Z_k = roles alive at stage k). + + (1) refined DYINGPAIR / generalised kill (SLACK PROOF C and H.2): `killed` + (2) Delta-Floor lemma (GAPLEMMA PROOF P1): kill if some 0 <= i <= d has |Delta_{i-1}| < 2i+4 - |U_i minus Delta| + (Delta = left roles of U_d absent from Z_K, K = d+1; U_i = roles entered by stage i). + +Expected (the counts quoted in README section 3): 117 layouts survive (1); (2) kills 102 of them; the 15 left are all c = 1, k = 1. +Controls: the plain B layout has no dying set; a pair entering at stage 0 is killed at i = 0 by (2); every pair layout that survives (1) +is killed by (2). Prints PASS_DFLOOR_COUNT.""" +L = 17; nl = 8 + +def two_arm(L=17, nl=8, hubs=2, grow=2): + masks = [set(range(L)) for _ in range(hubs)] + for k in range(nl): masks += [set(range(k, nl)) for _ in range(grow)] + for k in range(L - 1, nl - 1, -1): masks += [set(range(nl, k + 1)) for _ in range(grow)] + return masks + +def dying(specs): + ms = two_arm() + for arm, k, d, c, extra in specs: + assert arm == 'L' + full = set(range(k, nl)); new = set(range(k, d + 1)); ex = set(range(d + 1, nl)) + for _ in range(c): ms.remove(full); ms.append(set(new)) + for _ in range(extra): ms.append(set(ex)) + return ms + +def Z_count(ms, k): return sum(1 for m in ms if k in m) +def U(ms, j): return sum(1 for m in ms if min(m) <= j) +def left(ms): return [m for m in ms if max(m) < nl] + +def killed(ms, c, k, d): + K = d + 1; ZK = Z_count(ms, K); Dl = [m for m in left(ms) if min(m) <= d and K not in m] + old = ZK + len(Dl) - c <= 2 * K + 3 and all(U(ms, j - 1) <= 2 * j + 2 for j in range(k + 1, d + 1)) + ents = {min(m) for m in Dl} + new = len(ents) == 1 and 2 * K + 4 - ZK >= 1 and all(U(ms, t - 1) <= 2 * t + 2 for t in range(min(ents) + 1, d + 1)) + return old or new + +def dkill(ms, d): + K = d + 1; Dl = [m for m in left(ms) if min(m) <= d and K not in m] + for i in range(0, d + 1): + Uim = sum(1 for m in ms if min(m) <= i) + Dim1 = sum(1 for m in Dl if min(m) <= i - 1) + Dn = sum(1 for m in Dl if min(m) <= i) + if Dim1 < 2 * i + 4 - (Uim - Dn): return i + return None + +menu = []; surv = []; nk = 0; rem = [] +for c in (2, 1): + for k in (0, 1): + for d in range(max(k, 1), nl - 2): + for je in range(0, d + 1): + for eend in (d, nl - 1): + if k == 0 and je == 0: continue + ms = dying([('L', k, d, c, 0)]); ms.append(set(range(je, eend + 1))) + menu.append(1) + if killed(ms, c, k, d): continue + surv.append((c, k, d, je, eend)) + if dkill(ms, d) is None: rem.append((c, k, d, je, eend)) +print('menu', len(menu), 'survivors of the refined DYINGPAIR / generalised kill', len(surv), + 'killed by Delta-Floor', len(surv) - len(rem), 'remaining', len(rem)) +assert len(surv) == 117 and len(rem) == 15 +assert all(c == 1 and k == 1 for c, k, d, je, eend in rem) +assert all(c == 1 for c, *_ in rem) and not any(c == 2 for c, *_ in rem) # every pair layout dies +# controls +assert dying([]) == two_arm() +ms = dying([('L', 0, 3, 1, 0)]); ms.append(set(range(1, 4))) +assert dkill(ms, 3) == 0 # outermost entry, all own roles die: fails at i = 0 +ms = dying([('L', 1, 3, 1, 0)]); ms.append(set(range(0, 4))) +assert dkill(ms, 3) is None # family A: not killed by P1 (open case) +print('PASS_DFLOOR_COUNT') diff --git a/research/p104-findings-nov15/code/gaplemma/loc2.py b/research/p104-findings-nov15/code/gaplemma/loc2.py new file mode 100644 index 0000000000..8008576d82 --- /dev/null +++ b/research/p104-findings-nov15/code/gaplemma/loc2.py @@ -0,0 +1,52 @@ +"""GAPLEMMA loc2.py [N]: LOCAL relaxation of pair k=1 d=2 + slack 0..2 (K=3). +coords: H=0,1 E0=2,3 s=4 D=5,6 E2=7,8. stages 0,1,2 masks Z0={0..4}, Z1={0..6}, Z2={0..8}; units 2 per stage. +Constraints: x_i.y_i=0; Lemma S: y_i.x_j = sgn(i-j) w_i.z_j (i!=j), w_i.z_i=1; P3=T5..T0 block-diag wrt Delta={4,5,6}; +B = V3 cap R^Delta of dim 2 with sum-zero coefficients (Z^T beta=0): X beta1 = e_s, X beta2 = e_5 + u e_6.""" +import numpy as np, sys +from scipy.optimize import least_squares +n=9; Dl=[4,5,6]; Zr=[0,1,2,3,7,8] +masks=[list(range(5)),list(range(7)),list(range(9))] +sup=[masks[i//2] for i in range(6)] +offs=[];o=0 +for s_ in sup: offs.append(o); o+=2*len(s_) +nxy=o; dim=nxy+24+12+1 +def unpack(z): + X=np.zeros((n,6));Y=np.zeros((n,6)) + for i,s_ in enumerate(sup): + k=len(s_);X[s_,i]=z[offs[i]:offs[i]+k];Y[s_,i]=z[offs[i]+k:offs[i]+2*k] + W=z[nxy:nxy+12].reshape(6,2);Zm=z[nxy+12:nxy+24].reshape(6,2) + b1=z[nxy+24:nxy+30];b2=z[nxy+30:nxy+36];u=z[-1] + return X,Y,W,Zm,b1,b2,u +def res(z): + X,Y,W,Zm,b1,b2,u=unpack(z); r=[] + G=Y.T@X; K=W@Zm.T + for i in range(6): + for j in range(6): + if i==j: r.append(G[i,i]); r.append(K[i,i]-1) + else: r.append(G[i,j]-np.sign(i-j)*K[i,j]) + P=np.eye(n) + for i in range(6): P=(np.eye(n)-2*np.outer(X[:,i],Y[:,i]))@P + r+=list(P[np.ix_(Zr,Dl)].ravel())+list(P[np.ix_(Dl,Zr)].ravel()) + e=np.zeros(n);e[4]=1; r+=list(X@b1-e) + e=np.zeros(n);e[5]=1;e[6]=u; r+=list(X@b2-e) + r+=list(Zm.T@b1)+list(Zm.T@b2) + return np.array(r) +def diag(z): + X,Y,W,Zm,b1,b2,u=unpack(z); G=Y.T@X + t=[G[2*s,2*s+1]*G[2*s+1,2*s] for s in range(3)] + P=np.eye(n) + for i in range(6): P=(np.eye(n)-2*np.outer(X[:,i],Y[:,i]))@P + sv=np.linalg.svd(X,compute_uv=False) + return t,np.round(np.linalg.eigvals(P[np.ix_(Dl,Dl)]),6),sv[-1],np.linalg.matrix_rank(X[Zr],1e-8) +rng=np.random.default_rng(int(sys.argv[1]) if len(sys.argv)>1 else 0) +ok=0 +for trial in range(40): + z0=rng.normal(size=dim) + sol=least_squares(res,z0,method='trf',max_nfev=3000,xtol=1e-15,ftol=1e-15) + m=np.abs(res(sol.x)).max() + if m<1e-9: + ok+=1; t,ev,smin,rk=diag(sol.x) + print(trial,'res %.1e'%m,'t',np.round(t,3),'eigDelta',ev,'smin X %.2e'%smin,'rankXZ',rk) + np.save('loc2_sol%d.npy'%trial,sol.x) + else: print(trial,'fail %.1e'%m) +print('solved',ok) diff --git a/research/p104-findings-nov15/code/gaplemma/loc2f.py b/research/p104-findings-nov15/code/gaplemma/loc2f.py new file mode 100644 index 0000000000..529f06ae6f --- /dev/null +++ b/research/p104-findings-nov15/code/gaplemma/loc2f.py @@ -0,0 +1,19 @@ +"""GAPLEMMA loc2f.py [N]: loc2 + F1 enforced (Cx [X_<=2, e_H] = I_8, Cy [Y_<=2, e_H] = I_8) + t_s != -1 (u_s(1+t_s)=1).""" +import numpy as np, sys +exec(open('loc2.py').read().split('def diag')[0].replace('dim=nxy+24+12+1','dim=nxy+24+12+1+144+3')) +H=np.zeros((n,2));H[0,0]=H[1,1]=1 +base=nxy+37 +def res2(z): + r=list(res(z)); X,Y,W,Zm,b1,b2,u=unpack(z); G=Y.T@X + Cx=z[base:base+72].reshape(8,9); Cy=z[base+72:base+144].reshape(8,9); us=z[base+144:base+147] + r+=list((Cx@np.hstack([X,H])-np.eye(8)).ravel()); r+=list((Cy@np.hstack([Y,H])-np.eye(8)).ravel()) + for s in range(3): r.append(us[s]*(1+G[2*s,2*s+1]*G[2*s+1,2*s])-1) + return np.array(r) +rng=np.random.default_rng(int(sys.argv[1])); ok=0; best=[] +for trial in range(int(sys.argv[2])): + z0=rng.normal(size=dim) + sol=least_squares(res2,z0,method='trf',max_nfev=4000,xtol=1e-15,ftol=1e-15,gtol=1e-15) + m=np.abs(res2(sol.x)).max(); best.append(m) + print(trial,'%.2e'%m,flush=True) + if m<1e-9: ok+=1; np.save('loc2f_sol%d.npy'%trial,sol.x) +print('solved',ok,'best',min(best)) diff --git a/research/p104-findings-nov15/code/gaplemma/loc_A1_1.npy b/research/p104-findings-nov15/code/gaplemma/loc_A1_1.npy new file mode 100644 index 0000000000..13ecae437c Binary files /dev/null and b/research/p104-findings-nov15/code/gaplemma/loc_A1_1.npy differ diff --git a/research/p104-findings-nov15/code/gaplemma/loc_A2b2_7.npy b/research/p104-findings-nov15/code/gaplemma/loc_A2b2_7.npy new file mode 100644 index 0000000000..5e552e23e0 Binary files /dev/null and b/research/p104-findings-nov15/code/gaplemma/loc_A2b2_7.npy differ diff --git a/research/p104-findings-nov15/code/gaplemma/locgen.py b/research/p104-findings-nov15/code/gaplemma/locgen.py new file mode 100644 index 0000000000..5f90eb251b --- /dev/null +++ b/research/p104-findings-nov15/code/gaplemma/locgen.py @@ -0,0 +1,51 @@ +"""GAPLEMMA locgen.py [N]: generic LOCAL relaxation of the left arm up to stage K-1. +args: spec name, seed, trials. Constraints: x.y=0; Lemma S (W,Z); P_K block-diag wrt Delta; b vectors beta_i with +X[~Delta]beta_i=0, Z^T beta_i=0, beta^T beta = I; F1 (X and Y) for j=0..K-1 via C_j [X_<=j, e_H] = I; t_s != -1.""" +import numpy as np, sys +from scipy.optimize import least_squares +SPECS={ + # coords listed; masks per stage; Delta; b + 'A1': dict(n=7, masks=[[0,1,2,3,4],[0,1,2,3,4,5,6]], Dl=[4,5], b=1), # c=1 k=1 d=1 slack 0..1: H01 E0 23 s4 D5 e1 6 + 'A2': dict(n=9, masks=[[0,1,2,3,4],[0,1,2,3,4,5,6],list(range(9))], Dl=[4,5], b=1), # c=1 k=1 d=2 slack 0..2 (E2=7,8) + 'A2b2': dict(n=9, masks=[[0,1,2,3,4],[0,1,2,3,4,5,6],list(range(9))], Dl=[4,5], b=2), + 'P2': dict(n=9, masks=[[0,1,2,3,4],[0,1,2,3,4,5,6],list(range(9))], Dl=[4,5,6], b=2), # c=2 k=1 d=2 slack 0..2 (killed: control) +} +sp=SPECS[sys.argv[1]]; n=sp['n']; masks=sp['masks']; Dl=sp['Dl']; b=sp['b']; K=len(masks); NU=2*K +Zr=[i for i in range(n) if i not in Dl] +sup=[masks[i//2] for i in range(NU)] +offs=[];o=0 +for s_ in sup: offs.append(o); o+=2*len(s_) +nxy=o; o+=4*NU; ob=o; o+=b*NU; oc=o +csz=[(2*j+4)*(len(masks[j])) for j in range(K)] +o+=2*sum(csz); ot=o; o+=K; dim=o +H=np.zeros((n,2));H[0,0]=H[1,1]=1 +def unpack(z): + X=np.zeros((n,NU));Y=np.zeros((n,NU)) + for i,s_ in enumerate(sup): + k=len(s_);X[s_,i]=z[offs[i]:offs[i]+k];Y[s_,i]=z[offs[i]+k:offs[i]+2*k] + W=z[nxy:nxy+2*NU].reshape(NU,2);Zm=z[nxy+2*NU:nxy+4*NU].reshape(NU,2) + B=z[ob:ob+b*NU].reshape(NU,b) + return X,Y,W,Zm,B +def res(z): + X,Y,W,Zm,B=unpack(z); r=[]; G=Y.T@X; Kk=W@Zm.T + for i in range(NU): + for j in range(NU): + if i==j: r+=[G[i,i],Kk[i,i]-1] + else: r.append(G[i,j]-np.sign(i-j)*Kk[i,j]) + P=np.eye(n) + for i in range(NU): P=(np.eye(n)-2*np.outer(X[:,i],Y[:,i]))@P + r+=list(P[np.ix_(Zr,Dl)].ravel())+list(P[np.ix_(Dl,Zr)].ravel()) + r+=list((X[Zr]@B).ravel())+list((Zm.T@B).ravel())+list((B.T@B-np.eye(b)).ravel()) + o=oc + for M in (X,Y): + for j in range(K): + m=masks[j]; A=np.hstack([M[m][:,:2*j+2],H[m]]); C=z[o:o+csz[j]].reshape(2*j+4,len(m)); o+=csz[j] + r+=list((C@A-np.eye(2*j+4)).ravel()) + for s in range(K): r.append(z[ot+s]*(1+G[2*s,2*s+1]*G[2*s+1,2*s])-1) + return np.array(r) +rng=np.random.default_rng(int(sys.argv[2])); ok=0; best=[] +for trial in range(int(sys.argv[3])): + sol=least_squares(res,rng.normal(size=dim),method='trf',max_nfev=3000,xtol=1e-15,ftol=1e-15,gtol=1e-15) + m=np.abs(res(sol.x)).max(); best.append(m); print(trial,'%.2e'%m,flush=True) + if m<1e-9: ok+=1; np.save('loc_%s_%d.npy'%(sys.argv[1],trial),sol.x) +print('spec',sys.argv[1],'solved',ok,'of',len(best),'best %.2e'%min(best)) diff --git a/research/p104-findings-nov15/code/gaplemma/phantom_price.py b/research/p104-findings-nov15/code/gaplemma/phantom_price.py new file mode 100644 index 0000000000..ac31f337b0 --- /dev/null +++ b/research/p104-findings-nov15/code/gaplemma/phantom_price.py @@ -0,0 +1,54 @@ +"""GAPLEMMA phantom_price.py [C-param]: price family A (c=1 k=1 d, slack 0..d) as modelled (run6 caps) vs with the +dying single D made a PHANTOM (empty left mask: content only in thin units), lone-role rank 4 caps (PROOF E).""" +import sys, io, contextlib, os +_SLACK = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'slack'); os.chdir(_SLACK); sys.path.insert(0, _SLACK) +with contextlib.redirect_stdout(io.StringIO()): + import step1 +from icap import * +def pr(ms,r): + T,P,Z=invoice_iv(ms,r); return price(T,P,'',quiet=True) +B=pr(two_arm(),lambda a,b:4); g=lambda x:100*(min(p/q for p,q in zip(x,B))-1) +for d in range(1,6): + ms=step1.dying([('L',1,d,1,0)]); ms.append(set(range(0,d+1))) + a=g(pr(ms,lambda a,b:4)) + i=[j for j,m in enumerate(ms) if m==set(range(1,d+1))][0]; ms2=list(ms); ms2[i]=set() + try: b=g(pr(ms2,lambda a,b:4)) + except Exception as ex: b=float('nan'); print('err',ex) + print('d=%d modelled(rank4) %+.2f phantom D %+.2f'%(d,a,b)) +print('--- phantom case priced with run6 conservative caps but D excluded from u (inert D gives no constraint row):') +def Zs(ms): return [frozenset(i for i,m in enumerate(ms) if k in m) for k in range(L)] +for d in range(1,6): + ms=step1.dying([('L',1,d,1,0)]); ms.append(set(range(0,d+1))); Z=Zs(ms); idx=range(len(ms)) + Di=[j for j,m in enumerate(ms) if m==set(range(1,d+1))][0] + def mk(skipD): + def r(a,b): + ZI=frozenset.intersection(*Z[a:b+1]) + if any(Z[q]<=ZI or Z[q+1]<=ZI for q in range(a,b)): return 4 + Om=[i for i in idx if min(ms[i])<=a and i not in ZI and not (skipD and i==Di)] + return 4-min(2,len(Om)) + return r + print('d=%d run6-style caps %+.2f D inert (u counts s only) %+.2f'%(d,g(pr(ms,mk(False))),g(pr(ms,mk(True))))) +print('--- phantom case, PROVED caps (PROOF.md P3): rank 4 on every interval except a=0 across the death (rank >= 3):') +for d in range(1,6): + ms=step1.dying([('L',1,d,1,0)]); ms.append(set(range(0,d+1))); Z=Zs(ms) + si=len(ms)-1 + def r(a,b): + ZI=frozenset.intersection(*Z[a:b+1]) + if any(Z[q]<=ZI or Z[q+1]<=ZI for q in range(a,b)): return 4 + return 3 if (a==0 and si not in ZI) else 4 + print('d=%d phantom proved caps %+.2f'%(d,g(pr(ms,r)))) +print('--- family A, b=2 (open case), PROVED caps P4: rank >= 3 on every interval across the death (x~_0s != 0 kills the 2-drop):') +for d in range(1,6): + ms=step1.dying([('L',1,d,1,0)]); ms.append(set(range(0,d+1))); Z=Zs(ms) + def r(a,b): + ZI=frozenset.intersection(*Z[a:b+1]) + return 4 if any(Z[q]<=ZI or Z[q+1]<=ZI for q in range(a,b)) else 3 + print('d=%d b=2 caps r>=3 %+.2f'%(d,g(pr(ms,r)))) +print("--- family A' (slack survives, Delta={D}): run6 caps vs r=3") +for je in (0,1): + for d in range(1,6): + ms=step1.dying([('L',1,d,1,0)]); ms.append(set(range(je,nl))); Z=Zs(ms) + def r(a,b): + ZI=frozenset.intersection(*Z[a:b+1]) + return 4 if any(Z[q]<=ZI or Z[q+1]<=ZI for q in range(a,b)) else 3 + print("A' je=%d d=%d r=3 caps %+.2f"%(je,d,g(pr(ms,r)))) diff --git a/research/p104-findings-nov15/code/gaplemma/refac.py b/research/p104-findings-nov15/code/gaplemma/refac.py new file mode 100644 index 0000000000..e71ebe9f5a --- /dev/null +++ b/research/p104-findings-nov15/code/gaplemma/refac.py @@ -0,0 +1,42 @@ +"""GAPLEMMA refac.py [N]: REDUCTION test. Target T = diag(M', -I_Delta), M' = S'_1 S'_0 a generic 2-stage prefix on the 7 Z-coords +{H0,1 E0 2,3 e1 6 E2 7,8} with masks Z'_0, Z'_1 (arg). Find family-A d=2 left arm (stages on {0..4},{0..6},{0..8}; s=4, D=5) with +product T exactly (x.y=0). If solvable for generic M', the rest of the caller only sees M' and family A d=2 exists iff the +non-dying layout L' (first two stages replaced by S'_0,S'_1) has a root.""" +import numpy as np, sys +from scipy.optimize import least_squares +n=9; Zc=[0,1,2,3,6,7,8]; Dl=[4,5] +masks=[list(range(5)),list(range(7)),list(range(9))]; sup=[masks[i//2] for i in range(6)] +offs=[];o=0 +for s_ in sup: offs.append(o); o+=2*len(s_) +dim=o +def tv(x,y): return np.eye(len(x))-2*np.outer(x,y) +def iso_pair(rng,m): + while True: + x=np.zeros(n);y=np.zeros(n);x[m]=rng.normal(size=len(m));y[m]=rng.normal(size=len(m)); y-= (x@y)/(x@x)*x; return x,y +def target(rng,mk0,mk1): + M=np.eye(n) + for mk in (mk0,mk0,mk1,mk1): + x,y=iso_pair(rng,mk); M=tv(x,y)@M + T=M.copy(); T[np.ix_(Dl,Dl)]=-np.eye(2); return T +def unpack(z): + X=np.zeros((n,6));Y=np.zeros((n,6)) + for i,s_ in enumerate(sup): + k=len(s_);X[s_,i]=z[offs[i]:offs[i]+k];Y[s_,i]=z[offs[i]+k:offs[i]+2*k] + return X,Y +def prod(z): + X,Y=unpack(z);P=np.eye(n) + for i in range(6): P=tv(X[:,i],Y[:,i])@P + return P,X,Y +mode=sys.argv[1]; rng=np.random.default_rng(int(sys.argv[2])) +mk0={'A':[0,1,2,3],'B':[0,1,2,3,6],'C':Zc}[mode]; mk1=Zc +for tgt in range(int(sys.argv[3])): + T=target(rng,mk0,mk1); best=9 + def res(z): + P,X,Y=prod(z); return np.concatenate([(P-T).ravel(),np.sum(X*Y,0)]) + for st in range(int(sys.argv[4])): + sol=least_squares(res,rng.normal(size=dim),method='trf',max_nfev=4000,xtol=1e-15,ftol=1e-15,gtol=1e-15) + m=np.abs(res(sol.x)).max(); best=min(best,m) + if m<1e-10: + P,X,Y=prod(sol.x); G=Y.T@X; t=[G[2*s,2*s+1]*G[2*s+1,2*s] for s in range(3)] + print('target',tgt,'start',st,'SOLVED %.1e'%m,'t',np.round(t,3),flush=True); np.save('refac_%s_%d.npy'%(mode,tgt),np.concatenate([sol.x,T.ravel()])); break + else: print('target',tgt,'unsolved best %.1e'%best,flush=True) diff --git a/research/p104-findings-nov15/code/helperlb/analyze.py b/research/p104-findings-nov15/code/helperlb/analyze.py new file mode 100644 index 0000000000..4d6b84d1d1 --- /dev/null +++ b/research/p104-findings-nov15/code/helperlb/analyze.py @@ -0,0 +1,73 @@ +"""HELPERLB analyze.py (own code). Inputs needs.pkl (per-frame B_G on #7615), chain.py's LP rebuilt here per slice. +(1) slice validation: for a frame subset Sig, native chains restricted to stops in Sig (dropping stops never raises D) + must cost >= f(2)*sum_{G in Sig} B_G (free endpoints) and >= C2-LP(Sig) (census endpoints). +(2) slack location: per (a,b) class, per dim, per frame; marginal saving of each non-need stop. +(3) exact kappa of hypothetical invoices (PRICEKIT): helpers at the sound floor (681 single-stop helpers T[2],P[1]x2), + helpers all zero-cost (one hop), and the C2 optimum (flow decomposed into paths).""" +import pickle, math, collections, sys, json +sys.path.insert(0, '../../PRICEKIT'); sys.path.insert(0, '.') +import pricekit as pk +f = lambda r: r * math.log(r) if r > 0 else 0.0 +D = pickle.load(open('needs.pkl', 'rb')); out = D['out']; dimk = D['dimk']; chain = D['chain']; ends = D['ends'] +B = {G: o['B'] for G, o in out.items()}; need = {G for G in B if B[G] > 0} +def cost(p): return f(p[-1] - p[0]) - sum(f(p[i + 1] - p[i]) for i in range(len(p) - 1)) +def restricted(Sig): + tot = 0.0 + for h, ch in chain.items(): + a, b = ends[h]; tot += cost([a] + [dimk[x] for x in ch[1:-1] if x in Sig] + [b]) + return tot +full = restricted(set(out)); print('native helper D0/call %.4f' % full) +# ---- (1) per-frame and per-dim slices, free-endpoint floor +bad = 0 +for G in out: + if restricted({G}) + 1e-9 < f(2) * B[G]: bad += 1 +print('(1) per-frame slices: %d frames, violations %d' % (len(out), bad)) +for d in range(1, 9): + Sig = {G for G in out if dimk[G] == d} + print(' dim %d: B %4d floor %8.2f native-restricted %8.2f' % (d, sum(B[G] for G in Sig), f(2) * sum(B[G] for G in Sig), restricted(Sig))) +# ---- (2) slack: per class +cls = collections.defaultdict(lambda: [0, 0.0, 0.0, 0, 0]) +for h, ch in chain.items(): + a, b = ends[h]; c = cls[(a, b)]; c[0] += 1 + c[1] += cost([dimk[x] for x in ch]); c[2] += cost([a] + [dimk[x] for x in ch[1:-1] if x in need] + [b]) + c[3] += len(ch) - 2; c[4] += sum(1 for x in ch[1:-1] if x in need) +print('(2) class (a,b): n, native D, D need-stops-only, stops, need stops') +for k, c in sorted(cls.items(), key=lambda kv: -kv[1][1]): + if c[1] > 20: print(' %s n=%4d D=%8.1f need-only=%8.1f stops=%5d needstops=%5d' % (k, c[0], c[1], c[2], c[3], c[4])) +# per dim: stops vs B, and marginal saving of dropping each non-need stop (others kept) +st = collections.Counter(); nn = collections.Counter(); sav = collections.Counter(); savF = collections.Counter() +for h, ch in chain.items(): + p = [dimk[x] for x in ch] + for i in range(1, len(ch) - 1): + d = p[i]; st[d] += 1 + if ch[i] not in need: + nn[d] += 1; s = f(p[i + 1] - p[i - 1]) - f(p[i] - p[i - 1]) - f(p[i + 1] - p[i]); sav[d] += s; savF[ch[i]] += s +print('(2) dim: stops, B, excess, non-need stops, sum single-drop saving of non-need stops') +for d in range(1, 9): + Bd = sum(B[G] for G in out if dimk[G] == d) + print(' dim %d: stops %5d B %4d excess %5d nonneed %5d drop-saving %8.1f' % (d, st[d], Bd, st[d] - Bd, nn[d], sav[d])) +# excess at need frames (stops beyond B_G), top frames by excess +ex = sorted(((out[G]['P'] - B[G], G) for G in out), reverse=True)[:12] +print('(2) top frames by excess stops (P_G-B_G): ', [(G, dimk[G], out[G]['P'], B[G]) for e, G in ex]) +print(' zero-need frames: %d, their stops %d' % (sum(1 for G in out if B[G] == 0), sum(out[G]['P'] for G in out if B[G] == 0))) +# ---- (3) exact kappa of hypothetical invoices +W = pk.load('../../ATTACK/COURIER226/hit_c226_1.json.gz'); C = pk.load('../../MONOCALLER/7113_CALLER.json') +ci = pk.caller_invoice(C); L = ci['nJ']; nat = pk.native_from_word(W); inv = pk.invoice(ci, nat) +base = pk.price(inv, 20); K0 = float(base['kappa']['lo']); print('(3) control kappa', pk.kstr(base['kappa'], 14), 'D0', float(base['D0'])) +n = len(W['initial']); live = set(W['live']); H = W['hosted'] +main = live | {r + 112 for r in live} | set(H['X2']) | set(H['Y2']) +dim = [len(b) for b in W['frames']] +hT = collections.Counter(); hP = collections.Counter() +for r in range(n): + if r not in main: hT[dim[W['final'][r]] - dim[W['initial'][r]]] += L +for e in W['program']: + if e[0] == 'MOVE' and e[1] < n and e[1] not in main: hP[abs(dim[e[3]] - dim[e[2]])] += L +def kap(addT, addP): + T = collections.Counter(inv['T']); P = collections.Counter(inv['P']) + T.subtract(hT); P.subtract(hP); T.update(addT); P.update(addP) + T = {k: v for k, v in T.items() if v}; P = {k: v for k, v in P.items() if v} + pr = pk.price(dict(inv, T=T, P=P), 20); return float(pr['kappa']['lo']), float(pr['D0']) +k0, d0 = kap({}, {}); print(' helpers removed (all one-hop, D=0): kappa %s x%.4f D0/call %.2f' % ('%.10f' % k0, k0 / K0, d0 / L)) +nB = sum(B.values()); k1, d1 = kap({2: nB * L}, {1: 2 * nB * L}) +print(' helpers at sound floor (%d single stops): kappa %s x%.4f D0/call %.2f' % (nB, '%.10f' % k1, k1 / K0, d1 / L)) +pickle.dump(dict(cls=dict(cls), st=st, nn=nn, sav=sav, savF=savF, k0=k0, k1=k1), open('analyze.pkl', 'wb')) diff --git a/research/p104-findings-nov15/code/helperlb/chain.py b/research/p104-findings-nov15/code/helperlb/chain.py new file mode 100644 index 0000000000..e72c87ab12 --- /dev/null +++ b/research/p104-findings-nov15/code/helperlb/chain.py @@ -0,0 +1,89 @@ +"""CHAINBOUND chain.py (own code). Chain-cover bounds on #7615's helper-layer D0 given per-frame needs B_G (needs.pkl). +Helper cost (PRICEKIT native rule, monotone climb a=p0= k*f(2) for k interior stops (all chains in dims 0..9), so splitting into + singletons is never worse: min chain cover = f(2)*sum B_G (Dilworth adds nothing). +(2) conditional C2: the multiset of helper (start dim, end dim) pairs is #7615's (1,101 helpers), frames free. + Telescoping with fixed a: D(h) = sum_i g_a(p_i,p_{i+1}), g_a(p,q) = f(q-a)-f(p-a)-f(q-p) >= 0. + LP: one flow commodity per (a,b) class of value N_ab through need frames (a= B_G. Stops at non-need frames are dropped (removing a stop never raises D). + Integer chain covers are feasible LP points, so LP opt is a lower bound; certified via the dual (Lagrangian bound). +(3) reference C1: every helper {0}->V, count free (single commodity, a=0,b=9).""" +import pickle, math, itertools, collections, sys +import numpy as np +from scipy.optimize import linprog +from scipy.sparse import coo_matrix +f = lambda r: r * math.log(r) if r > 0 else 0.0 +D = pickle.load(open('needs.pkl', 'rb')); out = D['out']; keys = D['keys']; dimk = D['dimk'] +# ---- (1) exhaustive per-stop check +worst = 1e9 +for a in range(10): + for b in range(a + 2, 10): + mids = range(a + 1, b) + for k in range(1, b - a): + for S in itertools.combinations(mids, k): + p = [a, *S, b] + d = f(b - a) - sum(f(p[i + 1] - p[i]) for i in range(len(p) - 1)) + worst = min(worst, d - k * f(2)) +print('(1) min over all chains of D - k*f(2) = %.12f (>=0 required)' % worst) +sumB = sum(o['B'] for o in out.values()) +print('(1) free-endpoint chain-cover bound = f(2)*%d = %.4f' % (sumB, f(2) * sumB)) +# ---- need frames and strict inclusions +need = sorted(G for G, o in out.items() if o['B'] > 0) +mask = {G: sum(1 << x for x in keys[G]) for G in need} # span as bitset over 512 vectors +dim = {G: dimk[G] for G in need} +inc = [(G, H) for G in need for H in need if dim[G] < dim[H] and mask[G] & ~mask[H] == 0] +print('need frames', len(need), 'strict inclusions', len(inc)) +def solve(classes, label): + """classes: {(a,b): N}. Returns certified lower bound.""" + var = []; cost = []; ub = [] + for (a, b), N in classes.items(): + g = lambda p, q: f(q - a) - f(p - a) - f(q - p) + nodes = [G for G in need if a < dim[G] < b]; ns = set(nodes) + var.append(((a, b), 's', 't')); cost.append(0.0); ub.append(N) + for G in nodes: + var.append(((a, b), 's', G)); cost.append(g(a, dim[G])); ub.append(N) + var.append(((a, b), G, 't')); cost.append(g(dim[G], b)); ub.append(N) + for G, H in inc: + if G in ns and H in ns: + var.append(((a, b), G, H)); cost.append(g(dim[G], dim[H])); ub.append(N) + nv = len(var); print(label, 'vars', nv) + eq_rows = {}; er = []; ec = []; ev = []; beq = [] + def eqrow(k, rhs): + if k not in eq_rows: eq_rows[k] = len(beq); beq.append(rhs) + return eq_rows[k] + ur = []; uc = []; uv = []; bub = []; urow = {} + for G in need: + urow[G] = len(bub); bub.append(-out[G]['B']) + for j, (c, u, v) in enumerate(var): + if u == 's': r = eqrow((c, 's'), classes[c]); er.append(r); ec.append(j); ev.append(1.0) + else: r = eqrow((c, u), 0.0); er.append(r); ec.append(j); ev.append(-1.0) # out of u + if v != 't': + r = eqrow((c, v), 0.0); er.append(r); ec.append(j); ev.append(1.0) # into v + ur.append(urow[v]); uc.append(j); uv.append(-1.0) + Aeq = coo_matrix((ev, (er, ec)), shape=(len(beq), nv)).tocsr() + Aub = coo_matrix((uv, (ur, uc)), shape=(len(bub), nv)).tocsr() + c = np.array(cost); beq = np.array(beq); bub = np.array(bub); U = np.array(ub, float) + res = linprog(c, A_ub=Aub, b_ub=bub, A_eq=Aeq, b_eq=beq, bounds=list(zip([0] * nv, ub)), method='highs') + print(label, res.status, res.message[:60], 'opt %.4f' % res.fun) + y = res.eqlin.marginals; z = np.minimum(res.ineqlin.marginals, 0) + red = c - Aeq.T @ y - Aub.T @ z + LB = beq @ y + bub @ z + np.sum(np.minimum(red, 0) * U) + print(label, 'certified Lagrangian LB %.4f' % LB) + return res, LB, var +ends = D['ends']; cls = collections.Counter(ends.values()) +print('#7615 helper (start,end) dim census', sorted(cls.items())) +res2, LB2, var2 = solve(dict(cls), 'C2') +res1, LB1, var1 = solve({(0, 9): len(ends)}, 'C1(ref)') +# #7615's own value under each model: its chains restricted to need frames (a feasible point of C2) +ns = set(need); own = 0.0; full = 0.0 +for h, ch in D['chain'].items(): + a, b = ends[h]; p = [a] + [dimk[x] for x in ch[1:-1] if x in ns] + [b] + own += f(b - a) - sum(f(p[i + 1] - p[i]) for i in range(len(p) - 1)) + q = [dimk[x] for x in ch]; full += f(q[-1] - q[0]) - sum(f(q[i + 1] - q[i]) for i in range(len(q) - 1)) +print('#7615 helper D0 %.4f ; same chains with non-need stops dropped %.4f' % (full, own)) +# stop usage in C2 optimum +x = res2.x; byd = collections.Counter() +for j, (c, u, v) in enumerate(var2): + if v != 't' and x[j] > 1e-9: byd[c] += x[j] +print('C2 optimum: stops per class', {k: round(v, 2) for k, v in sorted(byd.items())}) +pickle.dump(dict(LB1=LB1, LB2=LB2, opt1=res1.fun, opt2=res2.fun, own=own, full=full, sumB=sumB), open('chain.pkl', 'wb')) diff --git a/research/p104-findings-nov15/code/helperlb/needs.py b/research/p104-findings-nov15/code/helperlb/needs.py new file mode 100644 index 0000000000..ebb615acca --- /dev/null +++ b/research/p104-findings-nov15/code/helperlb/needs.py @@ -0,0 +1,94 @@ +"""CHAINBOUND needs.py (own code; replay adapted from our HOSTLP imp.py). +Per frame G of #7615 with the fixed registers (mains + owned) and their events fixed: + V_G = pi-projected contents of fixed registers at every fixed event at G (arrival, departure, read by a fixed reg, + CLEAR, final residency, initial residency) R_G = rank V_G (mod 2^31-1) + n_G = # fixed arrivals into G that can carry content in SOME helper layer (initial residents + every fixed MOVE in; + INIT arrivals are exactly zero and are not counted) + m_G = # fixed departures out of G (every fixed MOVE out + every CLEAR + every final resident) + in_G = R_G - n_G (pass-through helpers needed to bring content in) + out_G = R_G - m_G (pass-through helpers needed to take content out) + B_G = max(0, in_G, out_G). +Also the #7615 interior helper stops per frame (soundness check).""" +import gzip, json, collections, pickle, sys +from fractions import Fraction +import flint +PR = 2147483647 +def fr(s): + f = Fraction(s); return (f.numerator % PR) * pow(f.denominator % PR, PR - 2, PR) % PR +W = json.load(gzip.open('../../ATTACK/COURIER226/hit_c226_1.json.gz')) +F = W['frames']; NP = W['public']; NO = W['owned_fields']; live = W['live']; H = W['hosted'] +mains = set(live) | {r + 112 for r in live} | set(H['X2']) | set(H['Y2']) +fixed = lambda r: r >= NP or r in mains +hel = [r for r in range(NP) if not fixed(r)]; hset = set(hel) +keep = [r for r in range(NP) if r not in hset]; col = {r: i for i, r in enumerate(keep)}; NC = len(keep) +def span(b): + s = {0} + for v in b: s |= {x ^ v for x in s} + return frozenset(s) +FS = [span(b) for b in F] +FK = {}; KEYS = [] # canonical frame id per span +for i, s in enumerate(FS): + if s not in FK: FK[s] = len(KEYS); KEYS.append(s) +fid = [FK[s] for s in FS] +dimk = [len(F[KEYS.index(s)]) if False else None for s in KEYS] +for i, s in enumerate(FS): dimk[fid[i]] = len(F[i]) +N = NP + NO +row = [{r: 1} if r < NP else {} for r in range(N)] +cur = list(W['initial']) + [None] * NO +def vec(r): + return {col[k]: c for k, c in row[r].items() if k in col} +V = collections.defaultdict(list); n = collections.Counter(); m = collections.Counter() +for r in range(NP): + if fixed(r): V[fid[cur[r]]].append(vec(r)); n[fid[cur[r]]] += 1 +def addto(d, src, c): + for k, v in src.items(): + nv = (d.get(k, 0) + c * v) % PR + if nv: d[k] = nv + else: d.pop(k, None) +for i, e in enumerate(W['program']): + op, r = e[0], e[1] + if op == 'MOVE': + if fixed(r): + a, b = fid[e[2]], fid[e[3]] + v = vec(r); V[a].append(v); V[b].append(v); m[a] += 1; n[b] += 1 + cur[r] = e[3]; continue + if op == 'INIT': row[r] = {}; cur[r] = e[2]; continue + if op == 'CLEAR': V[fid[e[2]]].append(vec(r)); m[fid[e[2]]] += 1; row[r] = {}; cur[r] = None; continue + if op == 'SCALE': + q = fr(e[2]); row[r] = {k: v * q % PR for k, v in row[r].items()}; continue + s = e[2]; c = fr(e[3]); f = fid[e[4]] + if fixed(s) and fixed(r): V[f].append(vec(s)); V[f].append(vec(r)) # contents at a fixed event (before) + if op == 'COPY': row[r] = {} + addto(row[r], row[s], c) + if fixed(s) and fixed(r): V[f].append(vec(r)) # and after +for r in range(NP): + if fixed(r): V[fid[cur[r]]].append(vec(r)); m[fid[cur[r]]] += 1 +def rank(vs): + vs = [v for v in vs if v] + if not vs: return 0 + M = flint.nmod_mat(len(vs), NC, PR) + for i, v in enumerate(vs): + for j, c in v.items(): M[i, j] = c + return M.rank() +# #7615 helper chains (frame ids) and interior stops +chain = {h: [fid[W['initial'][h]]] for h in hel} +for e in W['program']: + if e[0] == 'MOVE' and e[1] in chain: chain[e[1]].append(fid[e[3]]) +P = collections.Counter() +for h, ch in chain.items(): + for x in ch[1:-1]: P[x] += 1 +out = {} +for G in set(V) | set(P): + R = rank(V[G]); B = max(0, R - n[G], R - m[G]) + out[G] = dict(dim=dimk[G], R=R, n=n[G], m=m[G], inn=R - n[G], out=R - m[G], B=B, P=P[G]) +tot = lambda k: sum(max(0, o[k]) for o in out.values()) +import math +print('frames', len(out), 'sum in+', tot('inn'), 'sum out+', tot('out'), 'sum B', tot('B')) +print('floor 2ln2*sumB = %.4f' % (2 * math.log(2) * tot('B'))) +viol = [(G, o) for G, o in out.items() if o['P'] < o['B']] +print('violations on #7615', len(viol), viol[:5]) +print('B by dim', sorted(collections.Counter({o['dim']: 0 for o in out.values()}).items()) and + sorted((d, sum(o['B'] for o in out.values() if o['dim'] == d)) for d in range(10))) +print('frames out>in', sum(1 for o in out.values() if o['out'] > max(0, o['inn']))) +pickle.dump(dict(out=out, keys=[sorted(k) for k in KEYS], dimk=dimk, chain=chain, + ends={h: (dimk[ch[0]], dimk[ch[-1]]) for h, ch in chain.items()}), open('needs.pkl', 'wb')) diff --git a/research/p104-findings-nov15/code/helperlb/relay.py b/research/p104-findings-nov15/code/helperlb/relay.py new file mode 100644 index 0000000000..08347b2de6 --- /dev/null +++ b/research/p104-findings-nov15/code/helperlb/relay.py @@ -0,0 +1,13 @@ +"""HELPERLB relay.py: who uses the zero-need frames (B_G=0) - hop patterns, fixed-register presence.""" +import pickle, collections +D = pickle.load(open('needs.pkl', 'rb')); out = D['out']; dimk = D['dimk']; chain = D['chain'] +zero = {G for G, o in out.items() if o['B'] == 0} +pat = collections.Counter(); patz = collections.Counter() +for h, ch in chain.items(): + p = tuple(dimk[ch[i + 1]] - dimk[ch[i]] for i in range(len(ch) - 1)) + nz = sum(1 for x in ch[1:-1] if x in zero); pat[p] += 1; patz[p] += nz +print('zero-need stops by hop pattern (top):', [(p, patz[p], pat[p]) for p, _ in patz.most_common(8)]) +fx = sum(1 for G in zero if out[G]['R'] > 0 or out[G]['n'] > 0 or out[G]['m'] > 0) +print('zero-need frames %d; with any fixed register/event %d; pure helper-only %d' % (len(zero), fx, len(zero) - fx)) +byd = collections.Counter((dimk[G], out[G]['n'] > 0) for G in zero for _ in range(out[G]['P'])) +print('zero-need stops by (dim, fixed arrivals>0):', sorted(byd.items())) diff --git a/research/p104-findings-nov15/code/jointedit/cascade_t.py b/research/p104-findings-nov15/code/jointedit/cascade_t.py new file mode 100644 index 0000000000..46e39972f1 --- /dev/null +++ b/research/p104-findings-nov15/code/jointedit/cascade_t.py @@ -0,0 +1,32 @@ +# Depth-2 joint edit: main m's stop G is blocked by exactly one contact (partner p, lattice-locked: p's neighbour A +# not inside P / B not containing N). First edit p's neighbour stop Z in {A,B} (drop or re-stop, any legal edit with +# cost < +LIM), then re-scan m's stop G on the edited word. Keep pairs with total exact f-cost < 0. +import sys, pickle, jl, scan3, pricekit as pk +LIM = float(sys.argv[3]) if len(sys.argv)>3 else 0.05 +from collections import Counter; STAT=Counter() +W = pk.load(sys.argv[1]); M = jl.Word(W); out = []; tried = 0 +regs = sorted(M.mains) +for m in regs: + o = M.path(m) + for gi in range(1, len(o) - 1): + P, G, N = o[gi - 1], o[gi], o[gi + 1] + Gops = [i for i in M.ops[m] if M.tag[i] == G]; cache = {} + bl = [i for i in Gops if not scan3.opts_for(M, m, i, [P, N], M.ops[m], cache)] + STAT['bl%d'%min(len(bl),2)]+=1 + if len(bl) != 1: continue + e = M.ev[bl[0]]; p = e[2] if e[1] == m else e[1] + if p >= M.n: continue + po = M.path(p); k = po.index(G) + for Z in {po[k - 1] if k > 1 else None, po[k + 1] if k + 2 < len(po) else None} - {None}: + eds = scan3.scan_main(M, p, thresh=LIM, onlyG=Z); STAT['Z']+=1; STAT['Zlegal']+=bool(eds) + eds.sort(key=lambda r: r['d']) + for ed in eds[:6]: + tried += 1 + W1 = M.rebuild(ed['ov']); M1 = jl.Word(W1) + if G not in M1.path(m): continue + sec = scan3.scan_main(M1, m, thresh=-ed['d'] - 1e-9, onlyG=G) + for s2 in sec: + out.append(dict(m=m, G=G, p=p, Z=Z, e1=ed, e2=s2, d=ed['d'] + s2['d'])) + print('PAIR', m, G, 'partner', p, Z, 'd1 %.4f d2 %.4f tot %.4f' % (ed['d'], s2['d'], ed['d'] + s2['d']), flush=True) +print('tried', tried, 'pairs', len(out), dict(STAT)) +pickle.dump(out, open(sys.argv[2], 'wb')) diff --git a/research/p104-findings-nov15/code/jointedit/cnt.py b/research/p104-findings-nov15/code/jointedit/cnt.py new file mode 100644 index 0000000000..ede391b06c --- /dev/null +++ b/research/p104-findings-nov15/code/jointedit/cnt.py @@ -0,0 +1,13 @@ +import jl, scan3, pricekit as pk +from collections import Counter +W=pk.load('inputs/ATTACK/STACK7616/best7616.json.gz'); M=jl.Word(W) +for STAY in (False,True): + scan3.STAY=STAY; c=Counter() + for m in sorted(M.mains): + o=M.path(m) + for gi in range(1,len(o)-1): + P,G,N=o[gi-1],o[gi],o[gi+1]; Gops=[i for i in M.ops[m] if M.tag[i]==G]; cache={} + miss=sum(1 for i in Gops if not scan3.opts_for(M,m,i,[P,N],M.ops[m],cache)) + c[min(miss,3)]+=1 + c['stayopts']+=sum(1 for i in Gops for x in scan3.opts_for(M,m,i,[P,N],M.ops[m],cache) if x[4]) + print('STAY',STAY,sorted(c.items(),key=str)) diff --git a/research/p104-findings-nov15/code/jointedit/ctrl.py b/research/p104-findings-nov15/code/jointedit/ctrl.py new file mode 100644 index 0000000000..ae2a46089b --- /dev/null +++ b/research/p104-findings-nov15/code/jointedit/ctrl.py @@ -0,0 +1,27 @@ +import jl, scan2, pricekit as pk, sys +W=pk.load('inputs/ATTACK/STACK7616/best7616.json.gz'); M=jl.Word(W) +found=None +for m in sorted(M.mains): + o=M.path(m) + for P,N in zip(o,o[1:]): + Pops=[i for i in M.ops[m] if M.tag[i]==P] + if not Pops: continue + i=Pops[-1]; e=M.ev[i] + if e[0]!='ADD': continue + p=e[2] if e[1]==m else e[1] + if p>=M.n: continue + for G in range(len(M.sp)): + if M.legal(P,G) and M.legal(G,N): + ov={i:(M.key[i],G)} + if M.cost(M.path(m,ov)) is not None and M.cost(M.path(p,ov)) is not None: + found=(m,i,P,G,N,p); break + if found: break + if found: break +print('control edit',found) +m,i,P,G,N,p=found +W2=M.rebuild({i:(M.key[i],G)}) +r=pk.replay(W2); print('replay ok',r['ok'],r['problems'][:3]) +r0=pk.replay(W); print('rows equal', r['rows']==r0['rows']) +print('kappa ctrl',pk.kstr(jl.kappa(W2)['kappa'],14)) +M2=jl.Word(W2); res=scan2.scan_main(M2,m) +for x in res: print(x['m'],x['G'],x['P'],x['X'],x['N'],x['d'],x['slid']) diff --git a/research/p104-findings-nov15/code/jointedit/jl.py b/research/p104-findings-nov15/code/jointedit/jl.py new file mode 100644 index 0000000000..e7a8bdd31f --- /dev/null +++ b/research/p104-findings-nov15/code/jointedit/jl.py @@ -0,0 +1,137 @@ +# JOINTEDIT library: tag-sequence model of a word (non-MOVE events in order, each with a chart tag). +# A public register's path = clean([initial] + tags of its ops in order + [final]); MOVEs are regenerated from it. +# Joint edit = slide ops inside exact commutation windows + retag, so the main skips a stop; partners pay hop changes. +import sys, gzip, json, math, bisect +from collections import defaultdict +sys.path.insert(0, 'inputs/PRICEKIT'); import pricekit as pk +KAPPA = 0.0036576 +def f(r): return r ** (1 - KAPPA) if r > 0 else 0.0 +def spanmask(b): + S = {0} + for v in b: S |= {x ^ v for x in S} + m = 0 + for x in S: m |= 1 << x + return m +C_PATH = 'inputs/MONOCALLER/7113_CALLER.json' +_CI = None +def kappa(W): + global _CI + if _CI is None: _CI = pk.caller_invoice(pk.load(C_PATH)) + pr = pk.price(pk.invoice(_CI, pk.native_from_word(W)), 30) + return pr +class Word: + def __init__(s, W): + s.W = W; n = W['public']; s.n = n + s.sp = [spanmask(b) for b in W['frames']]; s.dim = [len(b) for b in W['frames']] + live = set(W['live']); H = W['hosted'] + s.mains = live | {r + 112 for r in live} | set(H['X2']) | set(H['Y2']) + s.ev = [e for e in W['program'] if e[0] != 'MOVE'] # E-order + s.key = [float(i) for i in range(len(s.ev))] + s.tag = [None] * len(s.ev) + s.ops = defaultdict(list) # public reg -> E idx (ADD/COPY/SCALE) + s.touch = defaultdict(list) # any reg -> E idx incl INIT/CLEAR + for i, e in enumerate(s.ev): + op = e[0] + if op in ('ADD', 'COPY'): + s.tag[i] = e[4]; regs = (e[1], e[2]) + elif op == 'SCALE': + s.tag[i] = e[3]; regs = (e[1],) + else: + s.tag[i] = e[2]; regs = (e[1],) + for r in regs: + s.touch[r].append(i) + if r < n and op != 'INIT' and op != 'CLEAR': s.ops[r].append(i) + def legal(s, a, b): return (s.sp[a] & ~s.sp[b]) == 0 and s.sp[a] != s.sp[b] + def path(s, r, override=None): + """override: dict E idx -> (key, tag) for moved/retagged ops (may include ops of other regs).""" + L = s.ops[r] + if override: + items = [] + for i in L: + k, t = override.get(i, (s.key[i], s.tag[i])); items.append((k, t)) + items.sort() + tags = [t for _, t in items] + else: + tags = [s.tag[i] for i in L] + q = [s.W['initial'][r]] + tags + [s.W['final'][r]] + o = [q[0]] + for c in q[1:]: + if c != o[-1]: o.append(c) + return o + def cost(s, o): + t = 0.0 + for a, b in zip(o, o[1:]): + if not s.legal(a, b): return None + t += f(abs(s.dim[b] - s.dim[a])) + return t + def conflicts(s, i, x): + """does event x fail to commute with ADD event i?""" + e = s.ev[i]; d, src = e[1], e[2]; y = s.ev[x]; op = y[0] + if op in ('INIT', 'CLEAR', 'SCALE'): return y[1] in (d, src) + # ADD/COPY y: dst y1, src y2 + if op == 'COPY' and y[1] in (d, src): return True + if y[2] == d: return True # reads d + if y[1] == src: return True # writes src + return False + def window(s, i): + e = s.ev[i] + if e[0] != 'ADD': return (i, i) + d, src = e[1], e[2] + cand = sorted(set(s.touch[d]) | set(s.touch[src])) + j = bisect.bisect_left(cand, i) + lo = -1 + for x in reversed(cand[:j]): + if s.conflicts(i, x): lo = x; break + hi = len(s.ev) + for x in cand[j + 1:]: + if s.conflicts(i, x): hi = x; break + return (lo, hi) + def rebuild(s, override): + """new word: apply override {E idx: (key, tag)}; regenerate MOVEs of affected public regs only.""" + W = s.W; n = s.n + aff = set() + for i in override: + e = s.ev[i] + for r in ((e[1], e[2]) if e[0] in ('ADD', 'COPY') else (e[1],)): + assert r < n, 'owned touched'; aff.add(r) + # new E order + keys = [override[i][0] if i in override else s.key[i] for i in range(len(s.ev))] + # attach old MOVEs (unaffected regs) to the next non-MOVE event in old order + prog = W['program']; att = defaultdict(list); tail = []; k = 0; pend = [] + for e in prog: + if e[0] == 'MOVE': + if e[1] in aff: continue + pend.append(e) + else: + if pend and k not in override: att[k].extend(pend); pend = [] + k += 1 + tail = pend + order = sorted(range(len(s.ev)), key=lambda i: (keys[i], i)) + evs = [] + for i in order: + e = list(s.ev[i]) + if i in override: + t = override[i][1] + if e[0] in ('ADD', 'COPY'): e[4] = t + elif e[0] == 'SCALE': e[3] = t + evs.append((i, e)) + # MOVE insertion for affected regs: before first op at each new chart + ins = defaultdict(list); endm = [] + for r in aff: + o = s.path(r, override) + opsr = sorted(s.ops[r], key=lambda i: (keys[i], i)) + first = {} + for i in opsr: + t = override[i][1] if i in override else s.tag[i] + first.setdefault(t, i) + for a, b in zip(o, o[1:]): + if b in first: ins[first[b]].append(['MOVE', r, a, b, False]) + else: endm.append(['MOVE', r, a, b, False]) + newp = [] + for i, e in evs: + newp.extend(att.get(i, [])) # NOTE: unaffected MOVEs keep their old successor event + newp.extend(ins.get(i, [])) + newp.append(e) + newp.extend(tail); newp.extend(endm) + W2 = dict(W); W2['program'] = newp + return W2 diff --git a/research/p104-findings-nov15/code/jointedit/law.py b/research/p104-findings-nov15/code/jointedit/law.py new file mode 100644 index 0000000000..0c44634d04 --- /dev/null +++ b/research/p104-findings-nov15/code/jointedit/law.py @@ -0,0 +1,27 @@ +# Why contacts at a main stop G cannot be re-hosted at P (main's prev) or N (main's next): partner neighbours A->G->B. +import jl, pricekit as pk +from collections import Counter +W=pk.load('inputs/ATTACK/STACK7616/best7616.json.gz'); M=jl.Word(W) +sp=M.sp; c=Counter(); single=Counter() +def nb(po,G): + k=po.index(G); return (po[k-1] if k>0 else None, po[k+1] if k+1=M.n: kinds.append('owned'); continue + po=M.path(p); A,B=nb(po,G) + Pok = A is None or (sp[A]&~sp[P])==0 # p's previous chart inside P + Nok = B is None or (sp[N]&~sp[B])==0 # N inside p's next chart + join = A is not None and (sp[A]|sp[P])==sp[G] # bitmask union of spans is not a span; check G = smallest span containing both + k=('P' if Pok else '')+('N' if Nok else '') + c['geo:'+(k or 'none')]+=1 + kinds.append('free' if k else 'geo') + bl=[k for k in kinds if k!='free'] + if len(bl)==1: single[bl[0]]+=1 +print(c); print('stops with exactly one geometrically blocked contact:',single) diff --git a/research/p104-findings-nov15/code/jointedit/law2.py b/research/p104-findings-nov15/code/jointedit/law2.py new file mode 100644 index 0000000000..270c92ca45 --- /dev/null +++ b/research/p104-findings-nov15/code/jointedit/law2.py @@ -0,0 +1,38 @@ +# Lock test: for each contact at main stop G (main P->G->N, partner A->G->B), the charts X where both can meet with +# all four neighbours kept are join(P,A) <= X <= meet(N,B). Forced iff join == meet == G. +import jl, pricekit as pk, pickle +from collections import Counter +W=pk.load('inputs/ATTACK/STACK7616/best7616.json.gz'); M=jl.Word(W) +sp=M.sp; fr=W['frames'] +def join(a,b): + S={0} + for v in fr[a]+fr[b]: + if v not in S: S|={x^v for x in S} + m=0 + for x in S: m|=1<0 else None, po[k+1] if k+1=M.n: locks.append('owned'); continue + A,B=nb(M.path(p),G) + lo = join(P,A) if A is not None else sp[P] + hi = (sp[N]&sp[B]) if B is not None else sp[N] + jf = lo==sp[G]; mf = hi==sp[G] + if A is not None and (sp[A]&~sp[P])==0 or B is not None and (sp[N]&~sp[B])==0 or B is None or A is None: k='free' + elif jf and mf: k='forced' + elif jf: k='join' # lower side forced, room above G + elif mf: k='meet' + else: k='room' + c[k]+=1; locks.append(k) + b=[k for k in locks if k!='free'] + st[(len(b),) if len(b)!=1 else (1,b[0])]+=1 +print('contacts',c); print('stops',sorted(st.items(),key=str)) diff --git a/research/p104-findings-nov15/code/jointedit/scan3.py b/research/p104-findings-nov15/code/jointedit/scan3.py new file mode 100644 index 0000000000..146d5c71c5 --- /dev/null +++ b/research/p104-findings-nov15/code/jointedit/scan3.py @@ -0,0 +1,125 @@ +# Generalised joint edit at a main stop G (main P->G->N): main becomes P->N (drop) or P->X->N (re-stop, X != G). +# Every m-op at G is re-hosted at P, X or N (monotone along the main's time order), each optionally slid inside its +# exact commutation window; partners pay their path change. Exact joint evaluation of all touched units. +import sys, pickle, jl, pricekit as pk +STAY = True +def nb(po, G): + k = po.index(G); return (po[k - 1] if k > 0 else None, po[k + 1] if k + 1 < len(po) else None) +def opts_for(M, m, i, charts, mlist, cache): + e = M.ev[i] + if e[0] == 'SCALE': return [(M.key[i], C, 0.0, None, None) for C in charts] + p = e[2] if e[1] == m else e[1] + if p >= M.n: return [] + if i not in cache: + lo, hi = M.window(i) if e[0] == 'ADD' else (i, i) + keys = {M.key[i]} + if e[0] == 'ADD': + keys.add(lo + 0.5) + for x in M.ops[p] + mlist: + if lo < x < hi and x != i: keys.add(x + 0.5) + cache[i] = (sorted(keys), M.cost(M.path(p)), {}) + keys, base, cc = cache[i] + out = [] + G = M.tag[i] + pst = [j for j in M.ops[p] if M.tag[j] == G and j != i] + third = set() + for j in pst: + ej = M.ev[j] + if ej[0] in ('ADD', 'COPY'): + q = ej[2] if ej[1] == p else ej[1] + third.add(q) + stay_ok = STAY and pst and m not in third and all(q < M.n for q in third) + for C in charts: + if C not in cc: + l = [] + for k in keys: + c = M.cost(M.path(p, {i: (k, C)})) + if c is not None: l.append((k, C, c - base, p, None)) + if stay_ok: + ov = {j: (M.key[j], C) for j in pst}; ov[i] = (k, C) + tot = 0.0 + for u in {p} | third: + c1 = M.cost(M.path(u, ov)) + if c1 is None: tot = None; break + tot += c1 - M.cost(M.path(u)) + if tot is not None: l.append((k, C, tot, p, ov)) + cc[C] = l + out += cc[C] + return out +def scan_main(M, m, thresh=-1e-9, onlyG=None): + o = M.path(m); mlist = M.ops[m]; sp = M.sp; res = [] + for gi in range(1, len(o) - 1): + P, G, N = o[gi - 1], o[gi], o[gi + 1] + if onlyG is not None and G != onlyG: continue + Gops = [i for i in mlist if M.tag[i] == G] + before = [i for i in mlist if i < Gops[0]]; after = [i for i in mlist if i > Gops[-1]] + prevnonP = max([i for i in before if M.tag[i] != P], default=-1) + lastP = before[-1] if before else -1 + firstN = after[0] if after else 10 ** 9 + nextnonN = min([i for i in after if M.tag[i] != N], default=10 ** 9) + # necessary containment for X from partners' neighbours (blocked contacts) + need_lo = sp[P]; need_hi = sp[N]; anyblocked = False; owned = False + for i in Gops: + e = M.ev[i] + if e[0] == 'SCALE': continue + p = e[2] if e[1] == m else e[1] + if p >= M.n: owned = True; break + if owned: continue + Xs = [None] + [c for c in range(len(sp)) if c != G and M.legal(P, c) and M.legal(c, N)] + cache = {} + for X in Xs: + charts = [P, N] if X is None else [P, X, N] + rank = {P: 0, N: 2} if X is None else {P: 0, X: 1, N: 2} + opts = []; ok = True + for i in Gops: + op = [x for x in opts_for(M, m, i, charts, mlist, cache) + if (x[1] == P and prevnonP < x[0] < firstN) or (x[1] != P and lastP < x[0] < nextnonN)] + if not op: ok = False; break + opts.append(op) + if not ok: continue + dmain = (jl.f(abs(M.dim[N] - M.dim[P])) if X is None else jl.f(M.dim[X] - M.dim[P]) + jl.f(M.dim[N] - M.dim[X])) \ + - jl.f(M.dim[G] - M.dim[P]) - jl.f(M.dim[N] - M.dim[G]) + if dmain + sum(min(x[2] for x in op) for op in opts) >= thresh: continue + ks = sorted(set(x[0] for op in opts for x in op)) + taus = [ks[0] - 1] + [(a + b) / 2 for a, b in zip(ks, ks[1:])] + [ks[-1] + 1] + best = None + for a in range(len(taus)): + for b in (range(a, len(taus)) if X is not None else [a]): + t1, t2 = taus[a], taus[b] + tot = 0.0; ch = [] + for op in opts: + c = [x for x in op if (rank[x[1]] == 0 and x[0] < t1) or (rank[x[1]] == 1 and t1 < x[0] < t2) or (rank[x[1]] == 2 and x[0] > t2)] + if not c: break + x = min(c, key=lambda x: x[2]); ch.append(x); tot += x[2] + else: + if X is not None and not any(x[1] == X for x in ch): continue + if best is None or tot < best[0]: best = (tot, ch) + if best is None or dmain + best[0] >= thresh: continue + ch = best[1]; ov = {} + units = {m} | {x[3] for x in ch if x[3] is not None} + for i, x in zip(Gops, ch): + if x[4]: + for j, v in x[4].items(): + if j in ov and ov[j] != v: ov = None; break + ov[j] = v + ej = M.ev[j]; units |= {ej[1]} | ({ej[2]} if ej[0] in ('ADD', 'COPY') else set()) + if ov is None: break + ov[i] = (x[0], x[1]) + if ov is None: continue + d = 0.0; bad = False + for r in units: + c1 = M.cost(M.path(r, ov)) + if c1 is None: bad = True; break + d += c1 - M.cost(M.path(r)) + if bad or d >= thresh: continue + res.append(dict(m=m, G=G, P=P, N=N, X=X, dmain=dmain, d=d, ov=ov, units=sorted(units), nops=len(Gops), + slid=sum(1 for i, x in zip(Gops, ch) if x[0] != M.key[i]))) + return res +if __name__ == '__main__': + W = pk.load(sys.argv[1]); M = jl.Word(W) + allr = [] + for m in sorted(M.mains): allr += scan_main(M, m) + allr.sort(key=lambda r: r['d']) + print('candidates', len(allr), 'drops', sum(1 for r in allr if r['X'] is None), 'slid', sum(1 for r in allr if r['slid'])) + for r in allr[:25]: print(r['m'], 'G', r['G'], 'P/X/N', r['P'], r['X'], r['N'], 'dmain %.4f d %.4f nops %d slid %d units %s' % (r['dmain'], r['d'], r['nops'], r['slid'], r['units'])) + pickle.dump(allr, open(sys.argv[2], 'wb')) diff --git a/research/p104-findings-nov15/code/jointedit/scanall.py b/research/p104-findings-nov15/code/jointedit/scanall.py new file mode 100644 index 0000000000..0fdae063e9 --- /dev/null +++ b/research/p104-findings-nov15/code/jointedit/scanall.py @@ -0,0 +1,6 @@ +import sys, pickle, jl, pricekit as pk; import scan3 as scan2 +W=pk.load(sys.argv[1]); M=jl.Word(W); allr=[] +for r in range(M.n): allr+=scan2.scan_main(M,r) +allr.sort(key=lambda r:r['d']); print('all-public candidates',len(allr)) +for r in allr[:20]: print(r['m'],'main' if r['m'] in M.mains else 'helper','G',r['G'],'P/X/N',r['P'],r['X'],r['N'],'d %.4f slid %d units %s'%(r['d'],r['slid'],r['units'])) +pickle.dump(allr,open(sys.argv[2],'wb')) diff --git a/research/p104-findings-nov15/code/mainsgap/b1.py b/research/p104-findings-nov15/code/mainsgap/b1.py new file mode 100644 index 0000000000..f87dadbdb5 --- /dev/null +++ b/research/p104-findings-nov15/code/mainsgap/b1.py @@ -0,0 +1,48 @@ +# B1: per-main exact best chain holding every partner's chart SET fixed (time relaxed). +import sys, pickle, math, itertools; sys.path.insert(0,'../../PRICEKIT'); import pricekit as pk +from collections import Counter, defaultdict +d=pickle.load(open('data.pkl','rb')); cls,stays,contacts,dim,frames=d['cls'],d['stays'],d['contacts'],d['dim'],d['frames'] +def span(b): + s={0} + for v in b: s|={x^v for x in s} + return frozenset(s) +sp=[span(b) for b in frames] +g=lambda r: r*math.log(r) if r>1 else 0.0 +def val(ch): return sum(g(dim[b]-dim[a]) for a,b in zip(ch,ch[1:])) +res={} +for m in sorted(cls): + st=stays[m]; path=[s[0] for s in st]; S,F=path[0],path[-1] + inI=lambda G: sp[S]<=sp[G]<=sp[F] + need=[] + for p in set(p for _,p,_,_ in contacts.get(m,[])): + Hp=set(s[0] for s in stays[p] if s[0] is not None and inI(s[0])) + assert Hp, (m,p) + if S in Hp or F in Hp: continue + need.append(frozenset(Hp)) + need=list(set(need)) + cand=sorted(set().union(*need) if need else set(), key=lambda G: dim[G]) + best=[None,-1] + # DFS over chains in candidate poset (strictly increasing subspaces), prune by upper bound + def ub(last, remaining): # best possible: one hop from last to F + return g(dim[F]-dim[last]) + def dfs(chain, uncovered, acc): + last=chain[-1] + if not uncovered: + v=acc+g(dim[F]-dim[last]) + if v>best[1]+1e-12: best[0]=chain+[F]; best[1]=v + return + if acc+ub(last,uncovered)<=best[1]+1e-12: return + # must cover the uncovered set containing the lowest-dim forced... branch on a set: pick uncovered set with fewest options above last + for G in cand: + if not (sp[last]r['actual']+1e-9) + if r["bound"]1 else 0.0 +def val(ch): return sum(g(dim[b]-dim[a]) for a,b in zip(ch,ch[1:])) +res={}; nodes=[0] +def solve(m, path_given=None): + st=stays[m]; path=[s[0] for s in st]; S,F=path[0],path[-1] + P={} + for p in set(p for _,p,_,_ in contacts.get(m,[])): + P[p]=[(G,s,e) for G,s,e in stays[p] if G is not None and sp[S]<=sp[G]<=sp[F]] + plist=list(P) + cand=sorted(set(G for p in plist for G,_,_ in P[p] if G not in (S,F)), key=lambda G: dim[G]) + best=[None,-1.0] + memo={} + def dfs(chain, a, unc, acc): + # at chart last, arrived at time a; unc = partners not yet served + nodes[0]+=1 + last=chain[-1] + if acc+g(dim[F]-dim[last])<=best[1]+1e-12: return + key=(last,a,unc) + if memo.get(key,-1e18)>=acc-1e-12: return + memo[key]=acc + here=[s for p in unc for (G,s,e) in P[p] if G==last and e>a and s>=a] + deps=sorted(set([a]+here)) # leave just after time d (d=a means immediately) + for dep in deps: + served=frozenset(p for p in unc if any(G==last and e>a and s<=dep and (sa or sa <=> s<=dep and e>a + served=frozenset(p for p in unc if any(G==last and s<=dep and e>a for G,s,e in P[p])) + rest=unc-served + # try finishing: jump to F at time dep + if all(any(G==F and e>dep for G,s,e in P[p]) for p in rest): + v=acc+g(dim[F]-dim[last]) + if v>best[1]+1e-12: best[0]=chain+[F]; best[1]=v + # feasibility of rest via later stops + for G in cand: + if not (sp[last]dep for p in rest for H,s,e in P[p]): continue + if any(not any((sp[G]<=sp[H]) and e>dep for H,s,e in P[p]) for p in rest): continue + dfs(chain+[G], dep, rest, acc+g(dim[G]-dim[last])) + dfs([S], -1, frozenset(plist), 0.0) + return best, path +if __name__=='__main__': + for m in sorted(cls): + best,path=solve(m) + res[m]=dict(cls=cls[m], actual=val(path), bound=best[1], chain=best[0], path=path) + if best[1]r['actual']+1e-9) + for k,a in sorted(agg.items()): print(k,'n',a[0],'actual %.1f B2 %.1f ideal %.1f improvable mains %d'%tuple(a[1:])) + print('nodes',nodes[0]) diff --git a/research/p104-findings-nov15/code/mainsgap/b3.py b/research/p104-findings-nov15/code/mainsgap/b3.py new file mode 100644 index 0000000000..f2ff5a135c --- /dev/null +++ b/research/p104-findings-nov15/code/mainsgap/b3.py @@ -0,0 +1,103 @@ +# B3: per-main exact best chain when the main must perform the SAME ADD multiset (exact arithmetic): +# - every other register's timed path and events are fixed; +# - each main contact may slide only inside its partner-side commutation window, and must happen at a chart +# where the partner sits during that window; +# - main-side order: recv-blocks / give-blocks / SCALE barriers keep their order (chart index along the chain is +# nondecreasing in block index). Time beyond that is relaxed (so B3 >= any exact single-main reroute). +import sys, pickle, math, bisect; sys.path.insert(0,'../../PRICEKIT'); import pricekit as pk +from collections import defaultdict +d=pickle.load(open('data.pkl','rb')); cls,stays,dim,frames=d['cls'],d['stays'],d['dim'],d['frames'] +W=pk.load('../../ATTACK/STACK7616/best7616.json.gz'); prog=W['program']; TT=len(prog) +def span(b): + s={0} + for v in b: s|={x^v for x in s} + return frozenset(s) +sp=[span(b) for b in frames] +g=lambda r: r*math.log(r) if r>1 else 0.0 +def val(ch): return sum(g(dim[b]-dim[a]) for a,b in zip(ch,ch[1:])) +read=defaultdict(list); write=defaultdict(list); mev=defaultdict(list) +for t,e in enumerate(prog): + op=e[0] + if op in('ADD','COPY'): write[e[1]].append(t); read[e[2]].append(t) + elif op=='SCALE': write[e[1]].append(t); read[e[1]].append(t) + elif op in('INIT','CLEAR'): write[e[1]].append(t); read[e[1]].append(t) +def window(t, lst): + i=bisect.bisect_left(lst,t); lo=lst[i-1] if i>0 else -1 + j=bisect.bisect_right(lst,t); hi=lst[j] if j p modified by us, non-commuting = p READ (incl. scale/init/clear); recv -> p WRITTEN + # (exclude this very event from the lists) + lst=[x for x in (read[p] if k=='give' else write[p]) if x!=t] + lo,hi=window(t,lst) + H=frozenset(G for G,s,e in stays[p] if G is not None and slo and sp[S]<=sp[G]<=sp[F]) + assert any(G==e2 for G in H for e2 in [prog[t][4]]), (m,t) + blocks[-1].append(H) + blocks=[sorted(set(b),key=len) for b in blocks if b] + return S,F,path,blocks +nodes=[0] +def solve(m): + S,F,path,blocks=build(m) + allH=set().union(*[set().union(*b) for b in blocks]) if blocks else set() + cand=sorted([G for G in allH if G not in (S,F)], key=lambda G: dim[G]) + best=[None,-1.0]; memo={} + def advance(G, j, rem): + # greedily assign at chart G: rem = remaining contacts (frozenset of H) of block j + while True: + rem=frozenset(H for H in rem if G not in H) + if rem: return j, rem + j+=1 + if j>=len(blocks): return j, frozenset() + rem=frozenset(blocks[j]) + def feasible_rest(G, j, rem): + # every remaining contact must have a chart strictly above G (necessary) + for H in rem: + if not any(sp[G]=acc-1e-12: return + memo[key]=acc + # finish at F? + j2,rem2=advance(F,j,rem) + if j2>=len(blocks): + v=acc+g(dim[F]-dim[last]) + if v>best[1]+1e-12: best[0]=chain+[F]; best[1]=v + for G in cand: + if not sp[last]=len(blocks): best=[[S,F],g(dim[F]-dim[S])] + else: dfs([S],j0,r0,0.0) + return best,path,len(blocks) +if __name__=='__main__': + res={} + for m in sorted(cls): + best,path,nb=solve(m) + res[m]=dict(cls=cls[m],actual=val(path),bound=best[1],chain=best[0],path=path,nblocks=nb) + if best[1]r['actual']+1e-9) + for k,a in sorted(agg.items()): print(k,'n',a[0],'actual %.1f B3 %.1f ideal %.1f improvable mains %d'%tuple(a[1:])) + print('nodes',nodes[0]) diff --git a/research/p104-findings-nov15/code/mainsgap/data.py b/research/p104-findings-nov15/code/mainsgap/data.py new file mode 100644 index 0000000000..e5e984b115 --- /dev/null +++ b/research/p104-findings-nov15/code/mainsgap/data.py @@ -0,0 +1,48 @@ +# Extract per-register chart paths (with time intervals) and per-main contacts from #7617. +import sys, pickle; sys.path.insert(0,'../../PRICEKIT'); import pricekit as pk +from collections import Counter, defaultdict +W=pk.load('../../ATTACK/STACK7616/best7616.json.gz') +n=W['public']; N=n+W['owned_fields']; H=W['hosted'] +cls={} +for r in W['live']: cls[r]='X'; cls[r+112]='Y' +for r in H['X2']: cls[r]='X2' +for r in H['Y2']: cls[r]='Y2' +dim=[len(b) for b in W['frames']] +cur=list(W['initial'])+[None]*W['owned_fields'] +stays=defaultdict(list) # reg -> list of [chart, t_in, t_out] +for r in range(n): stays[r].append([cur[r],-1,None]) +contacts=defaultdict(list) # main -> (t, partner, chart, role) +for t,e in enumerate(W['program']): + op=e[0] + if op=='MOVE': + r=e[1]; stays[r][-1][2]=t; stays[r].append([e[3],t,None]); cur[r]=e[3] + elif op=='INIT': + r=e[1]; stays[r].append([e[2],t,None]); cur[r]=e[2] + elif op=='CLEAR': + r=e[1]; stays[r][-1][2]=t; cur[r]=None + elif op in('ADD','COPY'): + r,s,f=e[1],e[2],e[4] + if r in cls: contacts[r].append((t,s,f,'recv')) + if s in cls: contacts[s].append((t,r,f,'give')) +T=len(W['program']) +for r in stays: + if stays[r][-1][2] is None: stays[r][-1][2]=T +pickle.dump(dict(cls=cls,stays=dict(stays),contacts=dict(contacts),dim=dim,frames=W['frames']),open('data.pkl','wb')) +# census +pc=Counter(); kinds=Counter(); mm=Counter() +for m in cls: + ps=set(p for _,p,_,_ in contacts.get(m,[])) + pc[cls[m]]+=len(ps) + for p in ps: kinds[(cls[m], cls.get(p,'own' if p>=n else 'H'))]+=1 + path=[dim[s[0]] for s in stays[m]] + mm[cls[m]]+=len(path)-1 +print('partners per class',pc); print('partner kinds',kinds); print('hops per class',mm) +print('contacts total',sum(len(v) for v in contacts.values())) +# contacts per chart position in main path (start/final/interior) +pos=Counter() +for m,cs in contacts.items(): + st=stays[m] + for (t,p,f,role) in cs: + i=[k for k,s in enumerate(st) if s[1]1 else 0 +gains=sorted(((r['bound']-r['actual'],m,r['cls']) for m,r in b2.items()),reverse=True) +tot=sum(x[0] for x in gains); print('B2 gain/call %.1f'%tot) +for k in (10,25,50,100,200): print(' top',k,'%.1f'%sum(x[0] for x in gains[:k])) +print(gains[:8]) diff --git a/research/p104-findings-nov15/code/mainsgap/variants.py b/research/p104-findings-nov15/code/mainsgap/variants.py new file mode 100644 index 0000000000..d78653fed0 --- /dev/null +++ b/research/p104-findings-nov15/code/mainsgap/variants.py @@ -0,0 +1,30 @@ +# Which constraint locks the B2 slack? B3w = windows only (blocks merged), B3o = block order only (full partner chart sets). +import sys, pickle, math +import b3 +from collections import defaultdict +mode=sys.argv[1] +orig=b3.build +def build2(m): + S,F,path,blocks=orig(m) + if mode=='w': + return S,F,path,[sorted(set().union(*[set(b) for b in blocks]),key=len)] if blocks else [] + # mode o: replace each contact's window set by the partner's full in-interval chart set: need partner ids -> rebuild + st=b3.stays[m]; ev=[] + for t,e in enumerate(b3.prog): + if e[0] in('ADD','COPY') and (e[1]==m)!=(e[2]==m): + ev.append(('recv' if e[1]==m else 'give', e[2] if e[1]==m else e[1])) + elif e[0]=='SCALE' and e[1]==m: ev.append(('scale',None)) + bl=[]; kind=None + for k,p in ev: + if k=='scale': bl.append([]); kind='scale'; continue + if k!=kind: bl.append([]); kind=k + bl[-1].append(frozenset(G for G,s,e in b3.stays[p] if G is not None and b3.sp[S]<=b3.sp[G]<=b3.sp[F])) + return S,F,path,[sorted(set(b),key=len) for b in bl if b] +b3.build=build2 +agg=defaultdict(lambda:[0,0]); res={} +for m in sorted(b3.cls): + best,path,nb=b3.solve(m); c=b3.cls[m] + agg[c][0]+=b3.val(path); agg[c][1]+=best[1]; res[m]=(b3.val(path),best[1],best[0],path) +pickle.dump(res,open('var_%s.pkl'%mode,'wb')) +for k,a in sorted(agg.items()): print(mode,k,'actual %.1f bound %.1f'%tuple(a)) +print('nodes',b3.nodes[0]) diff --git a/research/p104-findings-nov15/code/mainsgap/why.py b/research/p104-findings-nov15/code/mainsgap/why.py new file mode 100644 index 0000000000..9b4aa89d71 --- /dev/null +++ b/research/p104-findings-nov15/code/mainsgap/why.py @@ -0,0 +1,20 @@ +# For a main: which actual stops does B2 drop, and which contacts pin each dropped stop under B3 windows. +import sys, pickle, b3 +from collections import Counter +b2=pickle.load(open('b2.pkl','rb')) +def why(m, verbose=True): + S,F,path,blocks=b3.build(m); r=b2[m]; keep=set(r['chain']) + dropped=[G for G in path[1:-1] if G not in keep] + pins={} + for G in dropped: + # contacts whose window set allows NO chart of the B2 chain other than ... i.e. contacts pinned to charts not in the B2 chain + pins[G]=sum(1 for b in blocks for H in b if G in H and not (H & keep)) + if verbose: + print(m,b3.cls[m],'path dims',[b3.dim[x] for x in path],'B2 dims',[b3.dim[x] for x in r['chain']], + 'gain %.2f'%(r['bound']-r['actual']),'dropped stops -> #contacts pinned there (window set misses B2 chain):',{G:pins[G] for G in dropped}) + return pins +for m in (113,883,79,849,206): why(m) +tot=Counter(); n=Counter() +for m in b2: + for G,c in why(m,False).items(): tot[min(c,5)]+=1 +print('dropped stops by #pinned contacts (5=5+):',sorted(tot.items())) diff --git a/research/p104-findings-nov15/code/netc17/gate17.py b/research/p104-findings-nov15/code/netc17/gate17.py new file mode 100644 index 0000000000..df992bb78e --- /dev/null +++ b/research/p104-findings-nov15/code/netc17/gate17.py @@ -0,0 +1,78 @@ +"""NETC17 gate: distinct delivered pi-contents of a word + ratio-keyed greedy pair CSE (indexed). usage: gate17.py word.json.gz tag""" +import sys, collections, itertools, pickle, time, statistics as S +W5='inputs/' +sys.path.insert(0, W5+'REBUILDKIT') +from load import Word +from fractions import Fraction as Fr +P=2**61-1 +path,tag=sys.argv[1],sys.argv[2]; CAP=int(sys.argv[3]) if len(sys.argv)>3 else 40 +w=Word(path); W=w.W; n=w.n; N=w.N; dim=w.dim +Hs=set(r for r in range(n) if w.cls[r]=='h'); top=max(dim) +# a "helper" = class-h register that ever gives to a non-helper (carrier) or is read; census as NETC16 +rows=[({r:1} if rCAP) +inv=lambda x: pow(x,P-2,P) +def pairs(r): + ks=sorted(r) + for i,a in enumerate(ks): + ia=inv(r[a]) + for b in ks[i+1:]: yield (a,b,r[b]*ia%P) +idx=collections.defaultdict(set) +for i in sub: + for p in pairs(rows[i]): idx[p].add(i) +import heapq +heap=[(-len(v),p) for p,v in idx.items() if len(v)>=2]; heapq.heapify(heap) +nxt=10**9; steps=0; t0=time.time() +while heap: + negc,p=heapq.heappop(heap) + cur=idx.get(p,()) + if len(cur)<2: continue + if -negc!=len(cur): heapq.heappush(heap,(-len(cur),p)); continue + a,b,ratio=p; new=nxt; nxt+=1; steps+=1 + for i in list(cur): + r=rows[i] + for p2 in pairs(r): + s_=idx.get(p2) + if s_ is not None: s_.discard(i) + ca=r.pop(a); r.pop(b); r[new]=ca + for p2 in pairs(r): + idx[p2].add(i) + if len(idx[p2])>=2: heapq.heappush(heap,(-len(idx[p2]),p2)) + if steps%2000==0: print(' step',steps,'%.0fs'%(time.time()-t0),flush=True) +rem=sum(max(len(rows[i])-1,0) for i in sub) +naive=sum(len(rows[i])-1 for i in sub) +print(tag,'CSE steps',steps,'remaining adds',rem,'total ops',steps+rem,'(+ big-row naive %d)'%big_adds,'ratio naive/ops %.2f'%(naive/(steps+rem)),'helpers',nh) +pickle.dump(dict(nh=nh,contents=len(C),proj=len(R),naive=naive,ops=steps+rem,big=big_adds,hh=hhreads,gives=len(gives),givers=len(givers)),open('gate_%s.pkl'%tag,'wb')) diff --git a/research/p104-findings-nov15/code/netc17/key17.py b/research/p104-findings-nov15/code/netc17/key17.py new file mode 100644 index 0000000000..a57682df97 --- /dev/null +++ b/research/p104-findings-nov15/code/netc17/key17.py @@ -0,0 +1,49 @@ +"""NETC17 gate part 2: are delivered contents unique per (frame)? distinct (content,F) pairs vs helpers, native vs compiled.""" +import sys, collections, statistics as S +W5='inputs/' +sys.path.insert(0, W5+'REBUILDKIT') +from load import Word +from fractions import Fraction as Fr +P=2**61-1 +def run(path,tag): + w=Word(path); W=w.W; n=w.n; N=w.N; dim=w.dim + Hs=set(r for r in range(n) if w.cls[r]=='h'); top=max(dim) + rows=[({r:1} if r5,403 projective; hot content 2,953 gives = 53 V-mix helpers | +| 88 | Chain merge with giveq `m19.py`,`run19.py` | DONE (exact) | q=1 chain EXACT+accxfast; scaled distinct-receiver pair EXACT; scaled repeated-receiver chains FAIL V-mix | +| 89 | Exact builds q=1 chains 1/10/96 | DONE (exact) | D0 -13.7/-116.9/-548.0 vs control 40,558.2 (4.066x -> 4.011x), accxfast PASS all | +| 90 | Route-packing planner | NOT BUILT | ceiling 548 (q=1) to 1,757 (any q, partly unsettled), overlaps NETC15 | +""" +assert rows.count('\n')==7 and '| 90 |' in rows +open('FEATURES.md','w').write(rows) diff --git a/research/p104-findings-nov15/code/netc19/an19x.py b/research/p104-findings-nov15/code/netc19/an19x.py new file mode 100644 index 0000000000..a95db6b137 --- /dev/null +++ b/research/p104-findings-nov15/code/netc19/an19x.py @@ -0,0 +1,19 @@ +from m19 import * +N=load19(); H0=N.H0; U=units(N,H0) +g=groups(N,U) +res=[] +for c,idx in g.items(): + for ch in cover(N,U,idx): + if len(ch)<2: continue + h,qs=mk(N,H0,U,ch) + sav=sum(N.hcost(H0[i])[0] for i in ch)-N.hcost(h)[0] + res.append((sav,len(ch),ch,c)) +res.sort(key=lambda x:-x[0]) +print('chains',len(res),'units merged',sum(r[1] for r in res),'positive',sum(1 for r in res if r[0]>0),'total positive saving %.1f'%sum(r[0] for r in res if r[0]>0),'total all %.1f'%sum(r[0] for r in res)) +print('top',[(round(a,1),b) for a,b,_,_ in res[:15]]) +# hot content: keys 56,100 +for c,idx in g.items(): + ks=sorted(k for k,_ in c) + if ks==[56,100]: print('hot x56-x100: units',len(idx),'chains',len(cover(N,U,idx))) +base=sum(N.hcost(h)[0] for h in H0); print('base',base) +pickle.dump([(a,b,ch) for a,b,ch,_ in res],open('chains19x.pkl','wb')) diff --git a/research/p104-findings-nov15/code/netc19/cen19.py b/research/p104-findings-nov15/code/netc19/cen19.py new file mode 100644 index 0000000000..00ae9c9961 --- /dev/null +++ b/research/p104-findings-nov15/code/netc19/cen19.py @@ -0,0 +1,9 @@ +import sys,collections,pickle +exec(open('../NETC18/gate18.py').read().split("res={}")[0]) +nh,G,dim=census(W5+'NETC15/glob_p.json.gz') +cc=collections.Counter(g[1] for g in G) +top=cc.most_common(15) +for c,n in top: + fr=set(g[2] for g in G if g[1]==c); hs=set(g[0] for g in G if g[1]==c); rs=set(g[4] for g in G if g[1]==c) + print(n,'keys',len(c),sorted(c)[:3],'frames',len(fr),'helpers',len(hs),'recv',len(rs),flush=True) +pickle.dump((top,[g for g in G if g[1] in dict(top[:12])]),open('cen19.pkl','wb')) diff --git a/research/p104-findings-nov15/code/netc19/chainer.py b/research/p104-findings-nov15/code/netc19/chainer.py new file mode 100644 index 0000000000..d28f5983ec --- /dev/null +++ b/research/p104-findings-nov15/code/netc19/chainer.py @@ -0,0 +1,31 @@ +"""NETC11: greedy window chainer. plan_chains(N, P, hidx, maxend) -> list of chain dicts(hs, diffs).""" +from diffsolve import * + +def units(N, H0): + U = {} + for i, h in enumerate(H0): + b = h['block']; U[i] = dict(G=b['frame'], t1=b['t1'], r=b['r'], u=N.udeliv(h), blk=id(b)) + return U + +def plan_chains(N, P, H0, U, hidx, maxend=12, log=None): + hidx = sorted(hidx, key=lambda i: (U[i]['t1'], i)) + chains = []; cost = {i: N.hcost(H0[i])[0] for i in hidx} + sp = N.w.sp; tried = 0; ok = 0 + for i in hidx: + best = None + ends = [c for c in chains if U[c['hs'][-1]]['t1'] < U[i]['t1'] and sp[U[c['hs'][-1]]['G']] <= sp[U[i]['G']]] + ends.sort(key=lambda c: -U[c['hs'][-1]]['t1']) + for c in ends[:maxend]: + e = U[c['hs'][-1]]; tried += 1 + excl = {U[h]['r'] for h in c['hs']} + d = solve_diff(N, P, e, U[i], excl) + if d is None: continue + ok += 1 + new = N.merge(H0, c['hs'] + [i], c['diffs'] + [d]); cn = N.hcost(new)[0] + sav = c['cost'] + cost[i] - cn + if sav > 1e-9 and (best is None or sav > best[0]): best = (sav, c, d, cn) + if best is None: chains.append(dict(hs=[i], diffs=[], cost=cost[i])); continue + sav, c, d, cn = best + c['hs'].append(i); c['diffs'].append(d); c['cost'] = cn + if log: log('tried %d solved %d' % (tried, ok)) + return [c for c in chains if len(c['hs']) > 1] diff --git a/research/p104-findings-nov15/code/netc19/core11.py b/research/p104-findings-nov15/code/netc19/core11.py new file mode 100644 index 0000000000..948c60b01e --- /dev/null +++ b/research/p104-findings-nov15/code/netc19/core11.py @@ -0,0 +1,65 @@ +"""NETC11 (own code): chained-block carriers. Unit = one helper of one carrier (delivers u = L + sum(alpha picks) at +(t1, G) to r). Netc11 chains units h_1..h_m (different blocks, t1 ascending, G ascending) into ONE helper: alpha picks +of h_1, give; for j>=2 DIFFERENCE picks (sum = u_j - u_{j-1}, frames ascending in [G_{j-1}, G_j], times in +[t1_{j-1}, t1_j)), give; beta tail picks of h_m. Load = -beta(all picks) (as Netc.build); final pi = alpha(total).""" +import sys, os, pickle +W6 = 'inputs/' +sys.path[:0] = [W6 + 'NETC11', W6 + 'NETC6'] +import core6 +from core6 import * +import netc6, run_netc5 +from fractions import Fraction as Q +from decomp import tomod, ordered_solve +from netc import exact_solve +from width import Ech + +def vsum(snaps, vis): + tot = {} + for key, F, op in vis: + if op[0] == 'pick': + for kk, vv in snaps[op[1]][4].items(): + nv = tot.get(kk, 0) + op[2] * vv + if nv: tot[kk] = nv + else: tot.pop(kk, None) + return tot + +class Netc11(netc6.Netc6): + chains11 = () + def split(self, h): + t1 = h['block']['t1']; gk = (t1, 0.5) + a = [x for x in h['visits'] if x[2][0] == 'pick' and x[0] < gk] + z = [x for x in h['visits'] if x[2][0] == 'pick' and x[0] >= gk] + g = [x for x in h['visits'] if x[2][0] != 'pick']; assert len(g) == 1 + return a, g[0], z + def udeliv(self, h): + a, g, z = self.split(h) + u = dict(h['load']) + for kk, vv in vsum(self.snaps, a).items(): + nv = u.get(kk, 0) + vv + if nv: u[kk] = nv + else: u.pop(kk, None) + return u + def merge(self, H, hs, diffs): + first, last = H[hs[0]], H[hs[-1]] + a, g, _ = self.split(first); _, _, z = self.split(last) + vis = list(a) + [g] + for j in range(1, len(hs)): + vis += diffs[j - 1]; b = H[hs[j]]['block']; vis.append(((b['t1'], 0.5), b['frame'], ('give', b['r']))) + vis += z + L = {kk: -vv for kk, vv in self.beta(vsum(self.snaps, vis)).items()} + return dict(visits=vis, load=L, block=first['block']) + def build(self): + H = super().build() + self.H0 = H + if not self.chains11: return H + drop = set(); H = list(H) + for ch in self.chains11: + H[ch['hs'][0]] = self.merge(self.H0, ch['hs'], ch['diffs']); drop |= set(ch['hs'][1:]) + self.H = [h for i, h in enumerate(H) if i not in drop] + return self.H + +def load11(log=lambda *a: None): + run_netc5.Netc5 = Netc11 + N, nb = run_netc5.prepare(core6.BASE, log) + N.PB = [x.split(':') for x in os.environ.get('PB', '').split(',') if x] + return N diff --git a/research/p104-findings-nov15/code/netc19/m19.py b/research/p104-findings-nov15/code/netc19/m19.py new file mode 100644 index 0000000000..aa64f55b6a --- /dev/null +++ b/research/p104-findings-nov15/code/netc19/m19.py @@ -0,0 +1,27 @@ +import sys,os +sys.path.insert(0,'.') +from plan19x import * +import core11 +class Netc19(Netc14): + QS={} + def merge(self,H,hs,diffs): + first,last=H[hs[0]],H[hs[-1]] + a,g,_=self.split(first); _,_,z=self.split(last) + vis=list(a)+[g] + qs=self.QS[tuple(hs)] + for j in range(1,len(hs)): + b=H[hs[j]]['block']; vis.append(((b['t1'],0.5),b['frame'],('giveq',b['r'],qs[j]))) + vis+=z + L={kk:-vv for kk,vv in self.beta(vsum14(self.snaps,vis)).items()} + return dict(visits=vis,load=L,block=first['block']) +def load19(log=lambda *a:None): + run_netc5.Netc5=Netc19 + N,nb=run_netc5.prepare(core6.BASE,log) + N.PB=[x.split(':') for x in os.environ.get('PB','').split(',') if x] + return N +def uvec(U,i): return {k:Q(v) for k,v in U[i]['u'].items()} +def mk(N,H0,U,ch): + u1=uvec(U,ch[0]); qs=[Q(1)]+[ratio(u1,uvec(U,i)) for i in ch[1:]] + assert all(q is not None and q!=0 for q in qs) + N.QS[tuple(ch)]=qs + return N.merge(H0,ch,[[]]*(len(ch)-1)), qs diff --git a/research/p104-findings-nov15/code/netc19/plan19x.py b/research/p104-findings-nov15/code/netc19/plan19x.py new file mode 100644 index 0000000000..69375edbe0 --- /dev/null +++ b/research/p104-findings-nov15/code/netc19/plan19x.py @@ -0,0 +1,53 @@ +"""NETC19: unit-level route packing. Groups of units with the same PROJECTIVE delivered content (exact Fractions); +chain = t1 ascending, sp[G] nested ascending; helper gives with q = u_j/u_1. Min chain cover by bipartite matching.""" +import sys,os,collections,pickle,time +sys.path.insert(0,'.') +os.environ['PLAN']='inputs/NETC6/g2a.plan.pkl'; os.environ['GRAN']='862,1540' +from core15 import * +from chainer import units +from fractions import Fraction as Q +def ratio(u1,u2): + """u2 = q*u1 exactly? return q or None""" + if set(u1)!=set(u2) or not u1: return None + k=next(iter(u1)); q=Q(u2[k])/Q(u1[k]) + return q if all(Q(u2[x])==q*Q(u1[x]) for x in u1) else None +def groups(N,U): + g=collections.defaultdict(list) + for i,u in U.items(): + d=u['u'] + if not d: continue + g[frozenset((k,Q(v)) for k,v in d.items())].append(i) + return g +def cover(N,U,idx): + """min chain cover (list of chains, each list of unit idx) under t1 strict asc & sp nested""" + sp=N.w.sp; idx=sorted(idx,key=lambda i:(U[i]['t1'],i)); n=len(idx) + ok=lambda a,b: U[a]['t1']=0}; has_prev={j for j in range(n) if mt[j]>=0} + ch=[] + for i in range(n): + if i in has_prev: continue + c=[i] + while c[-1] in nxt: c.append(nxt[c[-1]]) + ch.append([idx[x] for x in c]) + return ch +if __name__=='__main__': + N=load14(); H0=N.H0; U=units(N,H0) + g=groups(N,U); print('units',len(U),'proj contents',len(g)) + tot=0;tot2=0 + rows=[] + for c,idx in g.items(): + ch=cover(N,U,idx); tot+=len(idx); tot2+=len(ch); rows.append((len(idx),len(ch),c)) + print('units in groups',tot,'chains',tot2) + rows.sort(key=lambda x:-x[0]) + print([(a,b) for a,b,_ in rows[:25]]) + # dim-window-only comparison (NETC18 style): number of distinct (content, G) pairs + pairs=sum(len(set(U[i]['G'] for i in idx)) for idx in g.values()); print('distinct (content,G) pairs',pairs) diff --git a/research/p104-findings-nov15/code/netc19/run19.py b/research/p104-findings-nov15/code/netc19/run19.py new file mode 100644 index 0000000000..0ac6a76990 --- /dev/null +++ b/research/p104-findings-nov15/code/netc19/run19.py @@ -0,0 +1,41 @@ +"""usage: NCH=k python run19.py OUT.json.gz : top-k route chains (identical projective content, giveq) from chains19.pkl, exact settle.""" +import os, sys, time, pickle +sys.path.insert(0,'.') +from m19 import * +import usweep14 +from run_netc5 import relay_phase +from settle6 import settle_and_check6 +t0=time.time(); log=lambda *a: print('[%5.0fs]'%(time.time()-t0),*a,flush=True) +N=load19(); H0=N.H0; U=units(N,H0) +for _k,_h in enumerate(H0): PIK[_k]=full_content(N,_h) +base=sum(N.hcost(h)[0] for h in H0); log('baseline plan cost %.1f helpers %d'%(base,len(H0))) +allc=pickle.load(open(os.environ.get('CF','chains19.pkl'),'rb')); k=int(os.environ['NCH']) +sel=[c for c in allc if c[0]>0][:k] +ch=[] +for sav,n_,hs in sel: + mk(N,H0,U,hs); ch.append(dict(hs=hs,diffs=[[]]*(len(hs)-1))) + log('chain units',len(hs),'frames',[U[i]['G'] for i in hs],'t1',[U[i]['t1'] for i in hs],'plan saving %.2f'%sav) +def setc(keep): + N.chains11=[ch[i] for i in sorted(keep)] + W1,lz=evaluate(N) + if not lz: + for spec in [x for x in os.environ.get('PB','').split(',') if x]: + o,td,tr,q,fr=spec.split(':'); N.add_pickback(int(o),int(td),int(tr),Q(q),int(fr)) + from dual import dual_scan as _ds + lz=_ds(N.emit())[0] + return W1,lz +keep=set(range(len(ch))); W1,lz=setc(keep) +log('chains %d merging %d units; cost %.1f helpers %d lossy %d'%(len(ch),sum(len(c['hs']) for c in ch),N.cost()[0],len(N.H),len(lz))) +if lz: + keys=sorted(keep); keep=set() + for kk in keys: + keep.add(kk) + if setc(keep)[1]: keep.discard(kk) + W1,lz=setc(keep); evaluate(N); log('DUAL repair kept %d of %d chains; cost %.1f lossy %d'%(len(keep),len(ch),N.cost()[0],len(lz))) +os.environ['PB']=','.join(':'.join(a) for a in N.PB) +W1,lz=relay_phase(N,log) +if lz: log('STOP: lossy after relays',lz); sys.exit(1) +log('plan cost after relays %.1f'%N.cost()[0]) +for i in sorted(keep): + h=ch[i]['hs']; log('KEPT chain',h[:3],'...',len(h)) +settle_and_check6(N,W1,sys.argv[1],log) diff --git a/research/p104-findings-nov15/code/netc19/sel19.py b/research/p104-findings-nov15/code/netc19/sel19.py new file mode 100644 index 0000000000..10ba47ace0 --- /dev/null +++ b/research/p104-findings-nov15/code/netc19/sel19.py @@ -0,0 +1,11 @@ +from m19 import * +import pickle +N=load19(); H0=N.H0; U=units(N,H0) +allc=pickle.load(open('chains19.pkl','rb')) +out=[] +for sav,n_,hs in allc: + if n_!=2 or sav<=0: continue + u1=uvec(U,hs[0]); q=ratio(u1,uvec(U,hs[1])) + if q!=1 and q>0 and U[hs[0]]['r']!=U[hs[1]]['r']: out.append((sav,n_,hs,q)) +print(len(out),out[:5]) +pickle.dump([(a,b,c) for a,b,c,_ in out[:1]],open('chains19s2.pkl','wb')) diff --git a/research/p104-findings-nov15/code/netc9/nat09.py b/research/p104-findings-nov15/code/netc9/nat09.py new file mode 100644 index 0000000000..5c98257718 --- /dev/null +++ b/research/p104-findings-nov15/code/netc9/nat09.py @@ -0,0 +1,69 @@ +"""NETC9 (own code): joint end-frame settlement graph of the native. For every native helper h that ends below V: +after h is pi-clean (tc), every op touching h with another HELPER p is an edge: supply p->h (h += c p) or sink h->p (p += c h). +Reports mutual-supply structure (SCCs, in/out degree, settle-frame multiplicities).""" +import sys, collections, pickle +W5='inputs/' +sys.path.insert(0, W5+'REBUILDKIT') +from fractions import Fraction as Q +from load import Word +w = Word(W5+'MAINFUSE/w7546.json.gz'); W=w.W; n=w.n; cls=w.cls; N=w.N; dim=w.dim +Hs=set(r for r in range(n) if cls[r]=='h'); top=max(dim) +V=[f for f in range(len(dim)) if dim[f]==top] +end={h:W['final'][h] for h in Hs} +low=sorted(h for h in Hs if dim[end[h]]tc[r]: + c='helper' if s in Hs else cls.get(s,'?') + nops[('in',c,'T' if s in T else '-')]+=1 + if s in Hs: sup[r].add(s) + efr[r][fr==end[r]]+=1 + if s in T and t>tc[s] and r!=s: + c='helper' if r in Hs else cls.get(r,'?') + nops[('out',c,'T' if r in T else '-')]+=1 + if r in Hs: snk[s].add(r) +print('post-clean ops:',sorted(nops.items(),key=lambda x:-x[1])) +# graph among low helpers: edge p->h if p supplies h's span +G=collections.defaultdict(set) +for h in low: + for p in sup[h]|snk[h]: + if p in T: G[h].add(p); G[p].add(h) +ud=sum(len(v) for v in G.values())//2 +print('undirected T-T edges',ud,'T nodes with any T-neighbour',sum(1 for h in low if G[h])) +# components +seen=set(); comps=[] +for h in low: + if h in seen or not G[h]: continue + st=[h]; seen.add(h); c=[] + while st: + x=st.pop(); c.append(x) + for y in G[x]: + if y not in seen: seen.add(y); st.append(y) + comps.append(c) +print('components (T-T)',sorted(map(len,comps),reverse=True)[:15],'singletons-with-no-T-edge',sum(1 for h in low if not G[h])) +print('suppliers per low helper (helper partners):',collections.Counter(min(len(sup[h]),9) for h in low)) +print('suppliers NOT in T (supplier ends at top or is non-low):',sum(1 for h in low for p in sup[h] if p not in T),'of',sum(len(sup[h]) for h in low)) +mut=sum(1 for h in low for p in sup[h] if p in T and h in sup[p]) +print('mutual supply pairs (h<->p both supply):',mut//2) +print('end-dim histogram',collections.Counter(dim[end[h]] for h in low)) +print('end frames distinct',len(set(end[h] for h in low)),'helpers per end frame',collections.Counter(collections.Counter(end[h] for h in low).values())) +pickle.dump(dict(low=low,tc=tc,sup={h:sorted(sup[h]) for h in low},snk={h:sorted(snk[h]) for h in low}),open('nat09.pkl','wb')) diff --git a/research/p104-findings-nov15/code/netc9/t_joint_bound.py b/research/p104-findings-nov15/code/netc9/t_joint_bound.py new file mode 100644 index 0000000000..89c03e0915 --- /dev/null +++ b/research/p104-findings-nov15/code/netc9/t_joint_bound.py @@ -0,0 +1,47 @@ +"""NETC9 joint-supply relaxation of NETC8 t_fbound_all (own code). MODE=A orig; B frames F<=E (any, other carriers at E supply their own picks); C = B and no lower validity bound (picks any time < t_r). +Original doc: G2 re-plan bound for F-MIX (own code, mod 2^61-1 screen): a TRIM-END carrier with F-MIX must retract its token from +the receiver r before r's next kept read, at a frame E that r occupies then (E != V). So its whole post-delivery tail +(-v) must come from snapshots at frames G <= F <= E valid inside (t1, t_r). Optimistic: chain order ignored, compiled +readers of r ignored, the tail costs one stop (E). Reports how many eligible carriers pass and the D0 they could save.""" +import sys, os, time, collections, math +sys.path.insert(0, 'inputs/NETC6') +from core6 import * +sys.path.insert(0, 'inputs/NETC7') +from run_netc5 import relay_phase +from settle import replay +from decomp import tomod, ordered_solve +import trim +MODE=os.environ.get('MODE','A') +t0 = time.time(); log = lambda *a: print('[%4.0fs]' % (time.time() - t0), *a, flush=True) +N = load(log); N.build(); W1, lz = relay_phase(N, lambda *a: None) +rows = replay(W1)[0]; n = N.w.n; V = N.V; Z = N.Z; bW = N.w.W; sp = N.w.sp; dim = N.w.dim +W0p0 = [y for y in N.W0 if not (y < n and bW['final'][y] == V)] +trim.choose(N, W1, rows, W0p0, log=log) +snaps = N.snaps; byframe = collections.defaultdict(list) +for i, s in enumerate(snaps): byframe[s[3]].append(i) +nf = len(bW['frames']); W0s = set(N.W0) +st = collections.Counter(); sv_ok = 0.0; sv_all = 0.0; okE = collections.Counter() +for i in (range(len(N.H)) if os.environ.get("ALL") else sorted(N._eligE)): + h = N.H[i]; b = h.get('block') + if b is None: st['no block'] += 1; continue + b = b; r, G, t1, v = b['r'], b['frame'], b['t1'], b['v'] + c0 = N.hcost(h)[0] + vis = sorted(h['visits'], key=lambda x: x[0]); pre = [x for x in vis if x[0][0] < t1 or (x[0][0] == t1 and x[0][1] <= 0.5)] + if r in W0s and bW['final'][r] != V: st['receiver W0p (pre-pay class)'] += 1; continue + nxt = min([t for t in N.reads[r] if t > t1] + [10 ** 9]) + Es = [] + for j, (t, fr, pi) in enumerate(N.traj[r]): + te = N.traj[r][j + 1][0] if j + 1 < len(N.traj[r]) else 10 ** 9 + if te <= t1 or t >= nxt or fr is None or fr == V: continue + if sp[G] <= sp[fr]: Es.append((dim[fr], fr, min(te, nxt))) + if not Es: st['r leaves for V / never at a frame >= G before its next read'] += 1; continue + found = None + for dE, E, tr in sorted(set(Es)): + C = [k for F in range(nf) if (sp[G] <= sp[F] if MODE=='A' else True) and sp[F] <= sp[E] and F != V for k in byframe[F] + if snaps[k][0] != r and snaps[k][1] < tr and (snaps[k][2] > t1 or MODE=='C')] + if C and ordered_solve(tomod(v), [tomod(snaps[k][4]) for k in C]) is not None: found = E; break + if found is None: st['tail -v not in window span'] += 1; continue + h2 = dict(h, visits=pre + [((t1, 0.9), found, ('pick', None, 0))], end=found) + s = c0 - N.hcost(h2)[0]; st['PASS'] += 1; sv_ok += s; okE[dim[found]] += 1 +log('eligible END carriers', len(N._eligE), dict(st)) +log('F-MIX re-plan bound: %d pass, optimistic saving %.1f D0/call (of 40,351.8); end dims %s' % (st['PASS'], sv_ok, dict(okE))) diff --git a/research/p104-findings-nov15/code/pricekit.py b/research/p104-findings-nov15/code/pricekit.py new file mode 100644 index 0000000000..10cd0ddf9c --- /dev/null +++ b/research/p104-findings-nov15/code/pricekit.py @@ -0,0 +1,490 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""PRICEKIT (our lane, vendored unchanged apart from this header): the one canonical exact pricer for P104 caller+native invoices. Standard library only. + +Accounting contract (README.md has the one-page version). Units: + rank r job a paid job of rank r (P[r] += 1) or a productive target of rank r (T[r] += 1). + S rank-mass surplus sum_r r (T_r - P_r) (= F(0)). + F(t) the bill sum_r (T_r - P_r) r^(1-t). + kappa the unique root of F on (0, hi]; certified by an interval enclosure (Decimal, error-bounded signs). + D0 sum (T-P) r ln r (= -F'(0); the HOSTCRIB/MAINRETIME/CALLERMAP "D", e.g. #6934 422589.15263423) + Dk sum (T-P) r^(1-kappa) ln r (= -F'(kappa); the DSPLIT/ARCHEQ dk.py/TRADELAW "D(kappa)") + +Charging rules (caller events, '#7113 format' = schema mixed-root-caller/1): + port count every caller event of type A/B is charged once per port of that type (len(types[t]['ports'])). + QUERY P[1] per port (thin unit / sign query). + IDLE_ONE s P[8|s|] per port (one idle over the stage set s). + IDLE_ODD s P[8|s|+1] per port. + LAMBDA ordinary work, 1000 units per port-use (#6913). lambda_mode='paid_rank1' charges P[1] per port + instead (counterfactual only). + ADD/SWAP/SCALE not invoiced (ordinary only). + diagonal row T[(q mod 2) + 8*sum(stage bits)] per port: a field of degree d is a target of rank 8d+1 (odd phase) + or 8d (even phase). Hubs are fields with d = #stages (r137 at 17 stages). + J (one stage) ONE native word per J event: stage = call = J event = one copy of the native word. The native word + serves all 104 ports internally, so it is NOT multiplied by ports. +Native word (schema p104-accumulated-block-composition/2): + paid P[|dim b - dim a|] per MOVE. + helpers T[dim final - dim initial] per public register that is not a main. + mains live X (r), live Y (r+112), hosted X2, Y2. A port with a hosted plane (r in hosted.Dports) has 4 + mains, otherwise 2. Mains are never in T: their climbs are paid (MOVEs) and their credit is the + caller diagonal. Ledger law: native mains per port per stage == caller slots covered per stage + (= mask rank - idle incidences); check_slots() tests it. +Parametric native (native_param): start from a word or a histogram native and + drop_mains_A/B k mains per A/B port per call removed, priced by main_unit: + 'rank1x8' (8 rank-1 jobs, the ARMS/MONOCALLER/SLOT3 unit), 'rank8' (one rank-8 job), or exact via + drop_registers() on a real word (preferred; this is not parametric). + c_A/c_B extra owned rank-1 paid jobs per A/B port per call (may be a Fraction). + dT/dP helper/paid histogram deltas per call. +Every switch is an explicit argument; nothing reads the environment. +""" +import gzip, json, math +from collections import Counter +from decimal import Decimal, localcontext +from fractions import Fraction as Q + +VERSION = 'pricekit/1 (2026-11-11)' +NPORTS = 104 + +# ------------------------------------------------------------------------------------------- io +def load(path): + op = gzip.open if str(path).endswith('.gz') else open + with op(path, 'rt') as f: + return json.load(f) + +def hist(d): + """dict with str/int keys -> Counter{int: int|Fraction}; zero entries dropped.""" + return Counter({int(k): (Q(v) if isinstance(v, str) else v) for k, v in d.items() if v}) + +def _add(dst, src, mult=1): + for r, v in src.items(): + dst[r] += mult * v + return dst + +# ------------------------------------------------------------------------------------------- native +def native_from_word(W, verify=False): + """Exact native invoice of one word (one call). verify=True also replays the word exactly (slow).""" + n = len(W['initial']); live = set(W['live']); H = W.get('hosted') or {} + X2, Y2 = list(H.get('X2', [])), list(H.get('Y2', [])) + main = live | {r + 112 for r in live} | set(X2) | set(Y2) + dim = [len(b) for b in W['frames']] + paid = Counter(); main_paid = Counter(); cls = {} + for r in live: cls[r] = 'X'; cls[r + 112] = 'Y' + for r in X2: cls[r] = 'X2' + for r in Y2: cls[r] = 'Y2' + by_class = {k: Counter() for k in ('X', 'Y', 'X2', 'Y2')} + for e in W['program']: + if e[0] == 'MOVE': + rk = abs(dim[e[3]] - dim[e[2]]); paid[rk] += 1 + if e[1] in cls: by_class[cls[e[1]]][rk] += 1 + helpers = Counter(dim[W['final'][r]] - dim[W['initial'][r]] for r in range(n) if r not in main) + dports = set(H.get('Dports', [])) + mains_per_port = {p: 2 + 2 * (p in dports and bool(X2)) for p in range(NPORTS)} + out = dict(T=+helpers, P=+paid, public=n, events=len(W['program']), mains=len(main), main_moves=by_class, + mains_per_port=mains_per_port, Dports=sorted(dports), source='word') + if verify: + out['replay'] = replay(W)['ok'] + return out + +def native_from_hist(paid, helpers, Dports=None, mains_A=4, mains_B=2): + dp = set(Dports) if Dports is not None else None + mpp = None if dp is None else {p: (mains_A if p in dp else mains_B) for p in range(NPORTS)} + return dict(T=hist(helpers), P=hist(paid), mains_per_port=mpp, Dports=sorted(dp) if dp else None, source='hist') + +def native_param(base, nA, nB, drop_mains_A=0, drop_mains_B=0, main_unit='rank1x8', c_A=0, c_B=0, dT=None, dP=None, + allow_negative=False): + """Parametric native per call. base = native dict. Returns a new native dict (source='param').""" + T = Counter(base['T']); P = Counter(base['P']) + unit = {'rank1x8': {1: 8}, 'rank8': {8: 1}}[main_unit] + for ports, k in ((nA, drop_mains_A), (nB, drop_mains_B)): + for r, v in unit.items(): + P[r] -= ports * k * v + P[1] += nA * Q(c_A) + nB * Q(c_B) + _add(T, dT or {}); _add(P, dP or {}) + mpp = base.get('mains_per_port') + if mpp is not None: + dp = set(base['Dports'] or []) + mpp = {p: m - (drop_mains_A if p in dp else drop_mains_B) for p, m in mpp.items()} + for r in list(P): # a negative native count is legal only if the caller absorbs it (invoice() re-checks) + if P[r] < 0 and not allow_negative: raise ValueError('native_param: negative paid count at rank %d' % r) + return dict(T=T, P=P, mains_per_port=mpp, Dports=base.get('Dports'), source='param', + param=dict(drop_mains_A=drop_mains_A, drop_mains_B=drop_mains_B, main_unit=main_unit, c_A=str(c_A), c_B=str(c_B))) + +# ------------------------------------------------------------------------------------------- exact word replay / pruning +def _span(b): + return frozenset(b) + +def replay(W): + """Exact replay (Fractions, sparse rows). Returns dict(ok, rows, frames, problems).""" + n = W['public']; N = n + W['owned_fields']; frames = W['frames'] + fs = [set(f) for f in frames] + rows = [{r: Q(1)} if r < n else {} for r in range(N)] + cur = list(W['initial']) + [None] * W['owned_fields']; owned = set(); bad = [] + for i, e in enumerate(W['program']): + op, r = e[0], e[1] + if op == 'MOVE': + a, b, priv = e[2:] + if cur[r] != a: bad.append(('move-from', i)); + cur[r] = b + elif op == 'INIT': + owned.add(r); rows[r] = {}; cur[r] = e[2] + elif op == 'CLEAR': + owned.discard(r); rows[r] = {}; cur[r] = None + elif op == 'SCALE': + q = Q(e[2]) + if cur[r] != e[3]: bad.append(('scale-frame', i)) + rows[r] = {k: v * q for k, v in rows[r].items()} + elif op in ('ADD', 'COPY'): + s = e[2]; q = Q(e[3]); f = e[4] + if cur[r] != f or cur[s] != f: bad.append(('add-frame', i)) + if op == 'COPY' and rows[r]: bad.append(('copy-nonzero', i)) + R = rows[r] + for k, v in rows[s].items(): + nv = R.get(k, 0) + q * v + if nv: R[k] = nv + else: R.pop(k, None) + else: + bad.append(('op', i, op)) + if owned: bad.append(('owned-alive', len(owned))) + if cur[:n] != list(W['final']): bad.append(('exit-frames',)) + return dict(ok=not bad, rows=rows, cur=cur, problems=bad[:20], nproblems=len(bad)) + +def drop_registers(W, drop): + """Exact dead-code deletion of the public registers in `drop` (e.g. hosted X2/Y2). + Keeps every event that some kept public output depends on; removes whole owned lifetimes with no live use. + Refuses (ValueError) if a dropped register feeds a kept output. Returns (W2, report).""" + drop = set(drop); n = W['public']; N = n + W['owned_fields']; PR = W['program'] + need_mask = 0 + dem = [0] * N + for r in range(n): + if r not in drop: dem[r] = 1 << r + keep = [False] * len(PR); feeds_dropped = 0 + for i in range(len(PR) - 1, -1, -1): + e = PR[i]; op, t = e[0], e[1] + if op in ('INIT', 'CLEAR'): + dem[t] = 0; continue + if op == 'MOVE': + continue + if dem[t]: + keep[i] = True + if op in ('ADD', 'COPY'): + s = e[2] + if s in drop: feeds_dropped += 1 + dem[s] |= dem[t] + if op == 'COPY': dem[t] = 0 + if feeds_dropped: + raise ValueError('drop_registers: %d kept events read a dropped register' % feeds_dropped) + # lifetime decisions for owned registers + life_keep = {}; open_life = {} + for i, e in enumerate(PR): + op, t = e[0], e[1] + if t >= n: + if op == 'INIT': open_life[t] = i; life_keep[i] = False + elif op in ('ADD', 'COPY', 'SCALE') and keep[i]: life_keep[open_life[t]] = True + if op in ('ADD', 'COPY') and keep[i] and e[2] >= n: life_keep[open_life[e[2]]] = True + life_of = {}; open_life = {} + for i, e in enumerate(PR): + op, t = e[0], e[1] + if t >= n: + if op == 'INIT': open_life[t] = i + life_of[i] = open_life[t] + out = []; dropped_moves = Counter(); dropped_lifetimes = 0; dim = [len(b) for b in W['frames']] + for i, e in enumerate(PR): + op, t = e[0], e[1] + if t in drop: + if op == 'MOVE': dropped_moves[abs(dim[e[3]] - dim[e[2]])] += 1 + continue + if t >= n: + if not life_keep[life_of[i]]: + if op == 'MOVE': dropped_moves[abs(dim[e[3]] - dim[e[2]])] += 1 + if op == 'INIT': dropped_lifetimes += 1 + continue + if op in ('INIT', 'CLEAR', 'MOVE') or keep[i]: out.append(e) + continue + if op == 'MOVE' or keep[i]: out.append(e) + W2 = dict(W); W2['program'] = out + H = dict(W.get('hosted') or {}) + if drop >= set(H.get('X2', [])) | set(H.get('Y2', [])) and H.get('X2'): + H['X2'] = []; H['Y2'] = []; H['dropped_hosted_plane'] = True + W2['hosted'] = H + rep = dict(events_before=len(PR), events_after=len(out), dropped_moves=dict(dropped_moves), + dropped_owned_lifetimes=dropped_lifetimes) + return W2, rep + +def zero_registers(W, drop): + """Hard deletion: registers in `drop` cease to exist (rows = 0). Every event that targets OR reads them is + removed, then dead code is pruned with drop_registers(). Only valid if verify_drop() shows every kept public row + unchanged; otherwise the word genuinely uses the dropped registers and a rebuild is needed.""" + drop = set(drop) + W1 = dict(W); W1['program'] = [e for e in W['program'] if not (e[1] in drop and e[0] != 'MOVE') + and not (e[0] in ('ADD', 'COPY') and e[2] in drop)] + removed_reads = sum(1 for e in W['program'] if e[0] in ('ADD', 'COPY') and e[2] in drop and e[1] not in drop) + W2, rep = drop_registers(W1, drop); rep['removed_reads'] = removed_reads + return W2, rep + +def verify_drop(W, W2, drop): + """Replay both words exactly; every kept public row and exit frame must be unchanged.""" + a = replay(W); b = replay(W2); n = W['public'] + same = all(a['rows'][r] == b['rows'][r] for r in range(n) if r not in set(drop)) + frames = all(b['cur'][r] == W['final'][r] for r in range(n)) + return dict(original_ok=a['ok'], pruned_ok=b['ok'], pruned_problems=b['problems'], kept_rows_equal=same, exit_frames=frames) + +# ------------------------------------------------------------------------------------------- caller +def caller_ports(C): + return {t: len(v['ports']) for t, v in C['types'].items()} + +def caller_invoice(C, lambda_mode='ordinary', only=None): + """Caller part of the invoice (no native). only='A'/'B' restricts to one port type. + Returns dict(T, P, nJ, ports, lambda_uses, ops).""" + assert lambda_mode in ('ordinary', 'paid_rank1') + ports = caller_ports(C); T = Counter(); P = Counter(); nJ = 0; lam = 0; ops = Counter() + for e in C['events']: + if e[0] == 'J': + nJ += 1; continue + ty, op = e[0], e[1] + if only and ty != only: continue + k = ports[ty]; ops[(ty, op)] += 1 + if op == 'QUERY': P[1] += k + elif op == 'IDLE_ONE': P[8 * len(e[3])] += k + elif op == 'IDLE_ODD': P[8 * len(e[3]) + 1] += k + elif op == 'LAMBDA': + lam += k + if lambda_mode == 'paid_rank1': P[1] += k + elif op in ('ADD', 'SWAP', 'SCALE', 'HOST'): pass + else: raise ValueError('caller op not in the alphabet: %r' % (op,)) + for ty, t in C['types'].items(): + if only and ty != only: continue + for dg in t['diagonal']: + T[int(dg[0]) % 2 + 8 * sum(dg[2:])] += ports[ty] + T[0] = 0; T = +T + return dict(T=T, P=+P, nJ=nJ, ports=ports, lambda_uses=lam, ops=ops) + +def check_slots(C, native=None): + """Per type and stage: mask rank, idle incidences, covered = mask - idles. With a native, compare covered with + the native's mains per port (the ledger law d = m).""" + nst = C['stages']; res = {} + for ty, t in C['types'].items(): + mask = [0] * (nst + 1); idle = [0] * (nst + 1) + for dg in t['diagonal']: + for k, b in enumerate(dg[2:], 1): + if b: mask[k] += 1 + for e in C['events']: + if e[0] == ty and e[1] in ('IDLE_ONE', 'IDLE_ODD'): + for k in e[3]: idle[k] += 1 + cov = [mask[k] - idle[k] for k in range(1, nst + 1)] + res[ty] = dict(mask=mask[1:], idle=idle[1:], covered=cov, header=sum(mask), idle_incidences=sum(idle)) + if native is not None and native.get('mains_per_port'): + mpp = native['mains_per_port'] + for ty, t in C['types'].items(): + ms = {mpp[p] for p in t['ports']} + res[ty]['native_mains'] = sorted(ms) + res[ty]['law_d_eq_m'] = (len(ms) == 1 and all(c == next(iter(ms)) for c in res[ty]['covered'])) + return res + +def caller_from_roots(fields, hosts_per_stage_cap, thin, ports, nst): + """Parametric caller (rule of our #6934 package, check.py outer()). fields = list of (mask stages, host stages); + a field of degree d is a target 8d+1; its unhosted mask stages are idled in maximal consecutive runs (P[8*run]); + a stage hosting more than hosts_per_stage_cap[k] fields pays one rank-8 idle per excess; thin rank-1 queries.""" + T = Counter(); P = Counter(); hc = Counter() + for ms, host in fields: + T[8 * len(ms) + 1] += ports + hs = set(host) + for k in hs: hc[k] += 1 + run = 0; prev = None + for k in sorted(ms): + if k in hs: + if run: P[8 * run] += ports + run = 0; prev = None; continue + if prev is not None and k == prev + 1: run += 1 + else: + if run: P[8 * run] += ports + run = 1 + prev = k + if run: P[8 * run] += ports + for k in range(nst): + if hc[k] > hosts_per_stage_cap[k]: P[8] += ports * (hc[k] - hosts_per_stage_cap[k]) + P[1] += ports * thin + return dict(T=+T, P=+P) + +# ------------------------------------------------------------------------------------------- invoice and price +def invoice(caller, native, calls=None): + """caller: dict from caller_invoice/caller_from_roots; native: native dict. calls defaults to caller nJ.""" + L = caller.get('nJ') if calls is None else calls + if not L: raise ValueError('invoice: number of native calls unknown (no J events and calls=None)') + T = Counter(caller['T']); P = Counter(caller['P']) + _add(T, native['T'], L); _add(P, native['P'], L) + for C_ in (T, P): + for r in list(C_): + if C_[r] == 0: del C_[r] + elif C_[r] < 0: raise ValueError('invoice: negative count at rank %d' % r) + return dict(T=T, P=P, calls=L) + +def S_of(T, P): + return sum(r * (T.get(r, 0) - P.get(r, 0)) for r in set(T) | set(P)) + +def _dec(x): + x = Q(x); return Decimal(x.numerator) / Decimal(x.denominator) + +def _coef(T, P): + return sorted((r, Q(T.get(r, 0)) - Q(P.get(r, 0))) for r in set(T) | set(P) if Q(T.get(r, 0)) != Q(P.get(r, 0))) + +def _F(cs, t, prec): + """F(t) and a rigorous bound on its evaluation error at this precision.""" + with localcontext() as c: + c.prec = prec; t = Decimal(t) + val = Decimal(0); mag = Decimal(0) + for r, k in cs: + p = (Decimal(r).ln() * (1 - t)).exp(); kd = _dec(k) + val += kd * p; mag += abs(kd) * p + err = mag * Decimal(10) ** (-(prec - 6)) + return val, err + +def _sign(cs, t, prec=60): + while prec <= 1200: + v, e = _F(cs, t, prec) + if abs(v) > e: return 1 if v > 0 else -1 + prec *= 2 + raise ArithmeticError('sign of F undetermined at t=%s' % t) + +def _unique(cs, hi, depth=14): + """Certify at most one root of F on [0, hi] by interval bounds (F excluded from 0, or F' < 0 on the piece).""" + with localcontext() as c: + c.prec = 40 + def pw(r, t): return (Decimal(r).ln() * (1 - Decimal(t))).exp() + def piece(a, b): + Fl = sum(_dec(k) * (pw(r, b) if k > 0 else pw(r, a)) for r, k in cs) + Fh = sum(_dec(k) * (pw(r, a) if k > 0 else pw(r, b)) for r, k in cs) + if Fl > 0 or Fh < 0: return 'noroot' + Dl = sum(_dec(k) * Decimal(r).ln() * (pw(r, b) if k > 0 else pw(r, a)) for r, k in cs if r > 1) + return 'mono' if Dl > 0 else 'unknown' + stack = [(Decimal(0), Decimal(hi), 0)]; mono = [] + while stack: + a, b, d = stack.pop(); s = piece(a, b) + if s == 'noroot': continue + if s == 'mono': mono.append((a, b)); continue + if d >= depth: return False + m = (a + b) / 2; stack += [(a, m, d + 1), (m, b, d + 1)] + mono.sort() + return all(mono[i][1] == mono[i + 1][0] for i in range(len(mono) - 1)) # one contiguous decreasing block + +def kappa(T, P, digits=50): + """Certified enclosure of the crossing. Returns dict(lo, hi, S, status). status NO_CROSSING if S <= 0.""" + cs = _coef(T, P); S = S_of(T, P) + if S <= 0: return dict(status='NO_CROSSING', S=S, lo=None, hi=None) + hi = Decimal('0.001') + while _sign(cs, hi) > 0: + hi *= 2 + if hi > 64: return dict(status='NO_SIGN_CHANGE', S=S, lo=None, hi=None) + lo = Decimal(0); uniq = _unique(cs, hi) + width = Decimal(10) ** (-digits - 2) + with localcontext() as c: + c.prec = digits + 30 + while hi - lo > width: + m = (lo + hi) / 2 + if _sign(cs, m, digits + 30) > 0: lo = m + else: hi = m + return dict(status='OK' if uniq else 'OK_UNIQUENESS_UNPROVED', S=S, lo=lo, hi=hi, unique=uniq) + +def Dsum(T, P, t=0, prec=60): + with localcontext() as c: + c.prec = prec; t = Decimal(t) + return sum(_dec(k) * Decimal(r).ln() * (Decimal(r).ln() * (1 - t)).exp() for r, k in _coef(T, P) if r > 1) + +def F_at(T, P, t, prec=60): + return _F(_coef(T, P), t, prec)[0] + +def price(inv, digits=50): + T, P = inv['T'], inv['P']; k = kappa(T, P, digits) + out = dict(S=S_of(T, P), paid_jobs=sum(P.values()), paid_positions=sum(r * v for r, v in P.items()), + targets=sum(T.values()), target_positions=sum(r * v for r, v in T.items()), kappa=k, + D0=Dsum(T, P, 0)) + if k.get('lo') is not None: out['Dk'] = Dsum(T, P, k['lo']) + return out + +def kstr(k, n=30): + if k.get('lo') is None: return k['status'] + return str(k['lo'])[:n + 2] + +def break_even_c(T, P, kappa_ref, ports_calls): + """Linear break-even: extra rank-1 paid per (port x call) at which kappa == kappa_ref. F(kref; c) = F0 - c*ports_calls.""" + return F_at(T, P, kappa_ref) / ports_calls + +# ------------------------------------------------------------------------------------------- ordinary units (#6934 package reading) +def word_ordinary(W): + """Per-call ordinary terms of a native word (allowance, reviewed upper, routes): our #6934 jclib.word_terms.""" + H = W['hosted']; nD = len(H['Dports']); n = len(W['initial']); live = set(W['live']) + main = live | {r + 112 for r in live} | set(H.get('X2', [])) | set(H.get('Y2', [])) + dim = [len(b) for b in W['frames']] + paid = Counter(abs(dim[e[3]] - dim[e[2]]) for e in W['program'] if e[0] == 'MOVE') + hel = Counter(dim[W['final'][r]] - dim[W['initial'][r]] for r in range(n) if r not in main) + life = sum(e[0] == 'INIT' for e in W['program']); allow = upper = 0 + for e in W['program']: + if e[0] in ('INIT', 'CLEAR'): allow += 2 + elif e[0] == 'SCALE': + q = Q(e[2]); a, d = abs(q.numerator), q.denominator + allow += 2 * (abs(a.bit_length() - d.bit_length()) + int(q < 0)); upper += 4 * (a.bit_length() + d.bit_length() + 1) + elif e[0] in ('ADD', 'COPY'): + q = Q(e[3]); a, d = abs(q.numerator), q.denominator + allow += 2 * (3 + a.bit_length() - 1 + bin(a).count('1') + d.bit_length() - 1); upper += 4 * (a.bit_length() + bin(a).count('1') + d.bit_length() + 1) + routes = (sum(paid.values()) + 2 * sum(hel.values()) + 208 + 2 * nD + 4 * 104 + 4 * nD + 4 * life) * 1000 + return dict(allow=allow, upper=upper, routes=routes, n=n, nD=nD) + +def _swaps(sel): + wanted = set(range(len(sel))); return len(list(zip(sorted(set(sel) - wanted), sorted(wanted - set(sel))))) + +def _shim(C, ty): + t = C['types'][ty]; off = t['offset']; ev = [] + for e in C['events']: + if e[0] == 'J': ev.append(['HOST', e[1]]) + elif e[0] == ty: + e = list(e[1:]) + if e[0] in ('ADD', 'SWAP', 'QUERY'): e[1] -= off; e[2] -= off + else: e[1] -= off + ev.append(e) + return dict(events=ev, roles=t['roles'], diagonal=t['diagonal']) + +def _rest(C, ports): + ev = C['events']; roles = C['roles']; mx = mn = mp = 0 + for role, (qp, zero, *sel) in enumerate(C['diagonal']): + axes = ([0] if qp else []) + [a for st, y in enumerate(sel, 1) if y for a in range(9 + 8 * (st - 1), 9 + 8 * st)] + k = _swaps(axes); mx += 2 * k * ports; mn += 2 * bool(qp) * ports; mp += int(qp == -1) * ports + ie = qn = qph = 0 + for e in ev: + if e[0] in ('IDLE_ONE',): + ie += 2 * _swaps([a for st in e[2] for a in range(9 + 8 * (st - 1), 9 + 8 * st)]) * ports + elif e[0] == 'QUERY': + qn += 2 * ports; qph += int(e[3] == -1) * ports + return (32 * roles * ports * len(ev) + sum(e[0] == 'QUERY' for e in ev) * 16 * ports + roles * ports * 26 + + (mx + mn + mp) * 1000 + ie * 1000 + (qn + qph) * 1000) + +def ordinary(C, W, odd_scale=1125): + """Raw ordinary readings of the #6934 package (literal/actual/upper/recharge/per_input). IDLE_ODD is not in that + formula (#7113-era callers): such callers get a flag, not a number for the IDLE_ODD routes.""" + t = word_ordinary(W); ports = caller_ports(C); L = sum(1 for e in C['events'] if e[0] == 'J') + rest = _rest(_shim(C, 'A'), ports['A']) + _rest(_shim(C, 'B'), ports['B']) + lam = sum(ports[e[0]] for e in C['events'] if e[0] != 'J' and e[1] == 'LAMBDA'); debit = 1000 * lam + sc = 4 * (odd_scale.bit_length() + 2) + 4 * (odd_scale.bit_length() + 2) + per_in = L * t['n'] * sc + lit = L * 10 ** 7 + rest; act = L * (t['allow'] + t['routes']) + rest; up = L * (t['upper'] + t['routes']) + rest + return dict(literal=lit, actual=act, upper=up, recharge=up + debit, per_input=up + per_in, per_input_recharge=up + per_in + debit, + lambda_uses=lam, lambda_debit=debit, idle_odd_present=any(e[0] != 'J' and e[1] == 'IDLE_ODD' for e in C['events'])) + +# ------------------------------------------------------------------------------------------- one-call convenience +def run(caller_path, native, lambda_mode='ordinary', digits=50): + C = load(caller_path); ci = caller_invoice(C, lambda_mode) + inv = invoice(ci, native); pr = price(inv, digits) + return dict(caller=ci, invoice=inv, price=pr, slots=check_slots(C, native)) + +def cite(label, pr): + """One-line citation string for RESULT.md files.""" + k = pr['kappa'] + return '[PRICEKIT %s] %s: S=%s kappa=%s D0=%.4f Dk=%s paid=%d' % ( + VERSION, label, pr['S'], kstr(k, 26), float(pr['D0']), ('%.8f' % float(pr['Dk'])) if 'Dk' in pr else '-', pr['paid_jobs']) + +if __name__ == '__main__': + import sys + if len(sys.argv) < 3: + print('usage: pricekit.py CALLER.json WORD.json[.gz] | NATIVE-RESULT.json'); sys.exit(1) + X = load(sys.argv[2]) + nat = native_from_word(X) if 'program' in X else native_from_hist(X['native']['paid'], X['native']['helpers']) + R = run(sys.argv[1], nat) + print(cite(sys.argv[2], R['price'])) diff --git a/research/p104-findings-nov15/code/relay/control.py b/research/p104-findings-nov15/code/relay/control.py new file mode 100644 index 0000000000..ed66c9cc30 --- /dev/null +++ b/research/p104-findings-nov15/code/relay/control.py @@ -0,0 +1,21 @@ +"""positive control: shear-transport a few NOT-pinned F-ops (and the movable SCALEs) and check final rows equal base (fastx k=4).""" +import sys, pickle, time +sys.path.insert(0, '../../NETCAUX'); import fastx +from mover import * +W = load(); NP, mains, fixed, dim = setup(W); P = W['program'] +X = pickle.load(open('explore.pkl', 'rb')); PN = pickle.load(open('pin.pkl', 'rb'))['nopath'] +base = [(float(i), e) for i, e in enumerate(P)]; tl = timeline(base, NP, W['owned_fields'], W['initial']) +R = fastx.proj_matrix(NP, 4, 1); a = fastx.FastReplay(W, R, 4) +done = 0; tried = 0 +for F in X['relay'][:120]: + for t, e in enumerate(P): + if done >= 4 or tried > 60: break + if evframe(e) != F or e[0] != 'ADD' or (e[1], e[2]) in PN.get(F, ()): continue + tried += 1 + j = t # base ops: index == key + for d in (-1, 1): + r = try_move(base, j, F, tl, d) + if r is None: continue + W2 = dict(W); W2['program'] = [x for k, x in r[0]]; b = fastx.FastReplay(W2, R, 4) + print('F', F, e, 'dir', d, '->', r[1], 'replay ok', b.ok, b.bad[:2], 'rows equal', a.proj == b.proj); done += 1; break +print('tried', tried, 'moved', done) diff --git a/research/p104-findings-nov15/code/relay/diag.py b/research/p104-findings-nov15/code/relay/diag.py new file mode 100644 index 0000000000..c9fe91535e --- /dev/null +++ b/research/p104-findings-nov15/code/relay/diag.py @@ -0,0 +1,28 @@ +import sys +from drive import * +base_ops = [(float(i), e) for i, e in enumerate(W['program'])] +tl = timeline(base_ops, NP, W['owned_fields'], W['initial']) +P = W['program'] +for F in map(int, sys.argv[1:]): + hs = {s[0] for s in X['stops'][F]} + for i, e in enumerate(P): + if evframe(e) != F or e[0] != 'ADD': continue + r, s = e[1], e[2] + print('F', F, 'op', i, e) + for d in (-1, 1): + # co-presence windows + st = stays(tl, r); ss = stays(tl, s) + w = [(max(a[0], b[0]), min(a[1], b[1]), a[2]) for a in st for b in ss if a[2] == b[2] and max(a[0], b[0]) < min(a[1], b[1]) and a[2] != F] + w = [x for x in w if (x[0] < i if d < 0 else x[1] > i)] + if not w: print(' dir', d, 'no co-presence'); continue + x = max(w, key=lambda x: x[1]) if d < 0 else min(w, key=lambda x: x[0]) + lo, hi = (min(x[1], i), i) if d < 0 else (i, max(x[0], i)) + blk = [] + for k in range(int(lo) + 1 if lo > -1e17 else 0, int(hi) if hi < 1e17 else len(P)): + O = P[k] + if O[0] == 'ADD' and (O[1] in (r, s) or O[2] in (r, s)): + kind = ('W' + ('r' if O[1] == r else 's') if O[1] in (r, s) else '') + ('R' + ('r' if O[2] == r else 's') if O[2] in (r, s) else '') + blk.append((k, kind, O[4], dim[O[4]])) + elif O[0] == 'SCALE' and O[1] in (r, s): blk.append((k, 'S' + ('r' if O[1] == r else 's'), O[3], dim[O[3]])) + from collections import Counter + print(' dir', d, 'meet frame', x[2], 'dim', dim[x[2]], 'intervening', len(blk), Counter(b[1] for b in blk), blk[:6]) diff --git a/research/p104-findings-nov15/code/relay/drive.py b/research/p104-findings-nov15/code/relay/drive.py new file mode 100644 index 0000000000..1573e19433 --- /dev/null +++ b/research/p104-findings-nov15/code/relay/drive.py @@ -0,0 +1,45 @@ +"""BIG-RELAY drive.py: frame-level removal of relay frames by shear transport; builds + verifies (fastx projections vs base).""" +import sys, pickle, time, gzip, json +sys.path.insert(0, '../../NETCAUX'); import fastx +from mover import * +W = load(); NP, mains, fixed, dim = setup(W) +X = pickle.load(open('explore.pkl', 'rb')) +def remove_frame(ops, F, stopsF, tl, verbose=False): + hs = {s[0] for s in stopsF} + for it in range(200): + idx = [i for i, (k, e) in enumerate(ops) if evframe(e) == F] + if not idx: break + moved = False + for j in idx: + e = ops[j][1] + if e[0] not in ('ADD', 'SCALE'): return None, 'nonadd' + for d in (-1, 1): + r = try_move(ops, j, F, tl, d) + if r is not None: + ops = r[0]; moved = True + if verbose: print(' moved', e, 'dir', d, 'to', r[1]) + break + if moved: break + if not moved: + if verbose: print(' stuck on', [ops[j][1] for j in idx]) + return None, 'stuck %d' % len(idx) + # merge MOVEs a->F->b + out = []; drop = set() + for s in stopsF: + h = s[0] + i1 = next(i for i, (k, e) in enumerate(ops) if e[0] == 'MOVE' and e[1] == h and e[3] == F) + i2 = next(i for i, (k, e) in enumerate(ops) if e[0] == 'MOVE' and e[1] == h and e[2] == F) + a = ops[i1][1][2]; drop.add(i1); e2 = list(ops[i2][1]); e2[2] = a; ops[i2] = (ops[i2][0], e2) + return [x for i, x in enumerate(ops) if i not in drop], 'ok' +def gainF(F): + return sum(f(dim[s[4]] - dim[s[3]]) - f(dim[F] - dim[s[3]]) - f(dim[s[4]] - dim[F]) for s in X['stops'][F]) +if __name__ == '__main__': + base_ops = [(float(i), e) for i, e in enumerate(W['program'])] + tl = timeline(base_ops, NP, W['owned_fields'], W['initial']) + for F in map(int, sys.argv[1:]): + t = time.time(); r, why = remove_frame(list(base_ops), F, X['stops'][F], tl, verbose=True) + print('F', F, why, 'gain %.4f' % gainF(F), '%.1fs' % (time.time() - t)) + if r is None: continue + W2 = dict(W); W2['program'] = [e for k, e in r] + R = fastx.proj_matrix(NP, 4, 1); a = fastx.FastReplay(W, R, 4); b = fastx.FastReplay(W2, R, 4) + print(' replay ok', b.ok, b.bad[:3], 'rows equal', a.proj == b.proj) diff --git a/research/p104-findings-nov15/code/relay/explore.py b/research/p104-findings-nov15/code/relay/explore.py new file mode 100644 index 0000000000..e1f0949bca --- /dev/null +++ b/research/p104-findings-nov15/code/relay/explore.py @@ -0,0 +1,40 @@ +import sys, collections, pickle +from lib import * +W = load(); NP, mains, fixed, dim = setup(W); P = W['program']; ch = chains(W) +# frames ever held by a fixed register (mains: their chains; owned: INIT/CLEAR/op frames) +fixedF = set() +for r in mains: + for fr, _ in ch[r]: fixedF.add(fr) +for e in P: + fr = evframe(e) + if fr is not None and (fixed(e[1]) or (e[0] in ('ADD','COPY') and fixed(e[2]))): fixedF.add(fr) +for r in range(NP): fixedF.add(W['final'][r]) if r in mains else None +stops = collections.defaultdict(list) # frame -> [(helper, arrive_idx, leave_idx, prev frame, next frame)] +for h in range(NP): + if h in mains: continue + c = ch[h] + for k in range(1, len(c) - 1): + stops[c[k][0]].append((h, c[k][1], c[k + 1][1], c[k - 1][0], c[k + 1][0])) +ops = collections.defaultdict(list) +for i, e in enumerate(P): + fr = evframe(e) + if fr is not None: ops[fr].append(i) +relay = sorted(F for F in stops if F not in fixedF) +print('frames with interior helper stops', len(stops), 'never-fixed', len(relay), 'stops there', sum(len(stops[F]) for F in relay)) +rows = [] +for F in relay: + S = stops[F]; hs = {s[0] for s in S} + o = [P[i] for i in ops[F]] + nadd = sum(1 for e in o if e[0] == 'ADD'); nsc = sum(1 for e in o if e[0] == 'SCALE') + inter = sum(1 for e in o if e[0] == 'ADD' and not (e[1] in hs and e[2] in hs)) + # gain if F removed from all stopping helpers (hop-cost only) + g = sum(f(dim[s[4]] - dim[s[3]]) - f(dim[F] - dim[s[3]]) - f(dim[s[4]] - dim[F]) for s in S) + # who stays simultaneous: max overlap + rows.append((F, dim[F], len(S), nadd, nsc, inter, g, len(ops[F]))) +rows.sort(key=lambda x: x[6]) +print('F dim nstops nadd nscale nonhh gain') +for r in rows[:15]: print(r) +print('dist nstops', collections.Counter(r[2] for r in rows)) +print('dist nadd', sorted(collections.Counter(min(r[3], 30) for r in rows).items())) +print('total gain all', sum(r[6] for r in rows)) +pickle.dump(dict(relay=relay, stops=dict(stops), rows=rows, fixedF=fixedF), open('explore.pkl', 'wb')) diff --git a/research/p104-findings-nov15/code/relay/lib.py b/research/p104-findings-nov15/code/relay/lib.py new file mode 100644 index 0000000000..a9684c0e61 --- /dev/null +++ b/research/p104-findings-nov15/code/relay/lib.py @@ -0,0 +1,22 @@ +"""BIG-RELAY lib (own code): load #7615, helper/fixed split, frame dims, per-frame event index.""" +import gzip, json, collections, math +BASE = 'inputs/ATTACK/COURIER226/hit_c226_1.json.gz' +def load(p=BASE): return json.load(gzip.open(p)) +f = lambda r: r * math.log(r) if r > 0 else 0.0 +def setup(W): + NP = W['public']; live = W['live']; H = W['hosted'] + mains = set(live) | {r + 112 for r in live} | set(H['X2']) | set(H['Y2']) + fixed = lambda r: r >= NP or r in mains + dim = [len(b) for b in W['frames']] + return NP, mains, fixed, dim +def chains(W): + ch = {r: [(W['initial'][r], -1)] for r in range(W['public'])} + for i, e in enumerate(W['program']): + if e[0] == 'MOVE' and e[1] < W['public']: ch[e[1]].append((e[3], i)) + return ch +def evframe(e): + if e[0] in ('ADD', 'COPY'): return e[4] + if e[0] == 'SCALE': return e[3] + if e[0] in ('INIT', 'CLEAR'): return e[2] + return None +def hcost(dims): return f(dims[-1] - dims[0]) - sum(f(dims[i + 1] - dims[i]) for i in range(len(dims) - 1)) diff --git a/research/p104-findings-nov15/code/relay/mover.py b/research/p104-findings-nov15/code/relay/mover.py new file mode 100644 index 0000000000..2006985cfa --- /dev/null +++ b/research/p104-findings-nov15/code/relay/mover.py @@ -0,0 +1,182 @@ +"""BIG-RELAY mover.py (own code): FRAME-LEVEL removal of a relay frame F by SHEAR TRANSPORT. +Each op at F is a shear/scale E on the stopping helpers. Moving E earlier (later) to a co-presence of its two registers at G != F +conjugates every intervening op O touching them: O' = E O E^-1 (earlier) / E^-1 O E (later). O' is decomposed into elementary +SCALE/ADD ops; legal iff every register it touches is at O's frame at O's time. When all F-ops are moved, the stopping helpers' +two MOVEs a->F->b merge into one a->b. Exact Fractions throughout.""" +from fractions import Fraction as Q +import bisect, collections, copy +from lib import * + +class Prog: + def __init__(self, W): + self.W = W; self.ops = [(float(i), e) for i, e in enumerate(W['program'])] + def keys(self): return [k for k, _ in self.ops] + +def timeline(ops, NP, NO, initial): + """per register: list of (key, frame) meaning frame from key on (initial key -inf).""" + tl = {r: [(-1e18, initial[r] if r < NP else None)] for r in range(NP + NO)} + for k, e in ops: + if e[0] == 'MOVE': tl[e[1]].append((k, e[3])) + elif e[0] == 'INIT': tl[e[1]].append((k - 1e-9, e[2])) + elif e[0] == 'CLEAR': tl[e[1]].append((k + 1e-9, None)) + return tl +def frame_at(tl, r, k): + L = tl.get(r) + if L is None: return None + i = bisect.bisect_right([x[0] for x in L], k) - 1 # small lists + return L[i][1] +def stays(tl, r): + L = tl[r]; return [(L[i][0], L[i + 1][0] if i + 1 < len(L) else 1e18, L[i][1]) for i in range(len(L))] + +def opmat(e): + """op -> (dict-of-dict N) with map s -> (I+N)s on register rows.""" + if e[0] == 'ADD': return {e[1]: {e[2]: Q(e[3])}} + if e[0] == 'SCALE': return {e[1]: {e[1]: Q(e[2]) - 1}} + return None +def mat_full(N, regs): + A = {r: {c: (Q(1) if r == c else Q(0)) for c in regs} for r in regs} + for r, row in N.items(): + for c, v in row.items(): A[r][c] += v + return A +def mul(A, B, regs): return {r: {c: sum(A[r][k] * B[k][c] for k in regs) for c in regs} for r in regs} +def decompose(A, regs): + """A (invertible, dict) -> list of elementary ops [(kind, r, s, q)] whose sequential application realises s -> A s. + Row-reduce A to I with row ops; A = E1^-1 ... Ek^-1; apply Ek^-1 first.""" + A = {r: dict(A[r]) for r in regs}; ops = [] # ops recorded as E_i applied to A from the left + regs = list(regs) + for c in regs: + if A[c][c] == 0: + p = next((r for r in regs if r != c and A[r][c] != 0 and all(A[r][cc] == (1 if r == cc else 0) or True for cc in regs)), None) + if p is None: return None + # row c += row p + for cc in regs: A[c][cc] += A[p][cc] + ops.append(('ADD', c, p, Q(1))) + if A[c][c] != 1: + q = 1 / A[c][c] + for cc in regs: A[c][cc] *= q + ops.append(('SCALE', c, c, q)) + for r in regs: + if r != c and A[r][c] != 0: + q = -A[r][c] + for cc in regs: A[r][cc] += q * A[c][cc] + ops.append(('ADD', r, c, q)) + # E_k ... E_1 A = I => A = E_1^-1 ... E_k^-1 ; applying A to s = apply E_k^-1 first + out = [] + for kind, r, s, q in reversed(ops): + if kind == 'SCALE': out.append(('SCALE', r, r, 1 / q)) + else: out.append(('ADD', r, s, -q)) + return out +def nontriv(A, regs): + return {r for r in regs for c in regs if A[r][c] != (1 if r == c else 0)} | {c for r in regs for c in regs if r != c and A[r][c] != 0} + +def conj(E, O, inv_first, regs): + """earlier: E O E^-1 ; later: E^-1 O E. E, O full dicts on regs.""" + Ei = inverse(E, regs) + return mul(mul(E, O, regs), Ei, regs) if not inv_first else mul(mul(Ei, O, regs), E, regs) +def inverse(E, regs): + seq = decompose(E, regs) # s -> E s = apply seq in order ; inverse = reversed inverted + I = mat_full({}, regs) + for kind, r, s, q in reversed(seq): + e = elem(kind, r, s, (1 / q) if kind == 'SCALE' else -q, regs) + I = mul(I, e, regs) if False else mul(I, e, regs) + # I currently = product applied... recompute safely by Gauss-Jordan instead + return gj_inverse(E, regs) +def elem(kind, r, s, q, regs): + return mat_full({r: {s: q - 1}} if kind == 'SCALE' else {r: {s: q}}, regs) +def gj_inverse(E, regs): + regs = list(regs); A = {r: dict(E[r]) for r in regs}; I = mat_full({}, regs) + for c in regs: + p = next(r for r in regs[regs.index(c):] if A[r][c] != 0) + if p != c: A[c], A[p] = A[p], A[c]; I[c], I[p] = I[p], I[c] + q = 1 / A[c][c]; A[c] = {k: v * q for k, v in A[c].items()}; I[c] = {k: v * q for k, v in I[c].items()} + for r in regs: + if r != c and A[r][c] != 0: + m = A[r][c]; A[r] = {k: A[r][k] - m * A[c][k] for k in regs}; I[r] = {k: I[r][k] - m * I[c][k] for k in regs} + return I + +def realise(A, regs, frame, key, tl): + """elementary op list for s -> A s at `frame`; None if a touched register is absent there at `key`.""" + t = nontriv(A, regs) + if any(frame_at(tl, r, key) != frame for r in t): return None + seq = decompose(A, sorted(t)) if t else [] + if seq is None: return None + out = [] + for kind, r, s, q in seq: + if kind == 'SCALE': + if q != 1: out.append(['SCALE', r, str(q), frame]) + elif q != 0: out.append(['ADD', r, s, str(q), frame]) + return out +def opframe(e): return evframe(e) + +def try_move(ops, j, F, tl, direction): + """ops: list of (key, op). Move op ops[j] (a shear/scale at F) earlier/later to a co-presence of its registers at G != F. + Returns new ops list or None.""" + k0, e = ops[j] + regsE = sorted({e[1], e[2]} if e[0] == 'ADD' else {e[1]}) + # co-presence windows of regsE at frames != F + st = [stays(tl, r) for r in regsE] + wins = [] + for a in st[0]: + for rest in [st[1:]]: + lo, hi, G = a + ok = True + for L in rest: + m = [(max(lo, x[0]), min(hi, x[1])) for x in L if x[2] == G and max(lo, x[0]) < min(hi, x[1])] + if not m: ok = False; break + lo, hi = m[0] + if ok and G != F and lo < hi: wins.append((lo, hi, G)) + if direction < 0: cand = sorted([w for w in wins if w[0] < k0], key=lambda w: -w[1]) + else: cand = sorted([w for w in wins if w[1] > k0], key=lambda w: w[0]) + for lo, hi, G in cand[:3]: + if direction < 0: + # place just before min(hi, k0): find insertion key strictly inside (lo, min(hi,k0)) + tgt = min(hi, k0); between = [i for i in range(len(ops)) if lo < ops[i][0] < k0 and i != j] + ins_key = None + # latest key in (lo, tgt) with no op of regsE after it until tgt: choose just below the first op >= tgt... simple: tgt - tiny + ins_key = nextafter_below(ops, tgt, lo) + if ins_key is None: continue + span = [i for i in range(len(ops)) if ins_key < ops[i][0] < k0 and i != j] + else: + tgt = max(lo, k0); ins_key = nextafter_above(ops, tgt, hi) + if ins_key is None: continue + span = [i for i in range(len(ops)) if k0 < ops[i][0] < ins_key and i != j] + if ins_key is None: continue + res = transport(ops, j, span, regsE, tl, direction) + if res is None: continue + newops = [x for i, x in enumerate(ops) if i != j and i not in res[0]] + for kk, op in res[1]: newops.append((kk, op)) + Ae = mat_full(opmat(e), regsE) + seq = realise(Ae, regsE, G, ins_key, tl) + if seq is None: continue + for n_, op in enumerate(seq): newops.append((ins_key + n_ * 1e-6 * (1 if direction > 0 else 1), op)) + newops.sort(key=lambda x: x[0]) + return newops, G + return None +def nextafter_below(ops, tgt, lo): + ks = [k for k, _ in ops if lo < k < tgt] + a = max(ks) if ks else lo + if a == -1e18: a = tgt - 1 + return (a + tgt) / 2 if tgt - a > 1e-3 else None +def nextafter_above(ops, tgt, hi): + ks = [k for k, _ in ops if tgt < k < hi] + b = min(ks) if ks else hi + if b == 1e18: b = tgt + 1 + return (tgt + b) / 2 - 1e-5 if b - tgt > 1e-3 else None + +def transport(ops, j, span, regsE, tl, direction): + """conjugate each op in span touching regsE. returns (set of replaced indices, [(key, newop)]) or None.""" + E = ops[j][1]; rem = set(); add = [] + for i in span: + k, O = ops[i] + if O[0] == 'MOVE' or O[0] in ('INIT', 'CLEAR'): continue + part = {O[1], O[2]} if O[0] in ('ADD', 'COPY') else {O[1]} + if not (part & set(regsE)): continue + if O[0] == 'COPY': return None + regs = sorted(part | set(regsE)) + Em = mat_full(opmat(E), regs); Om = mat_full(opmat(O), regs) + C = conj(Em, Om, direction > 0, regs) + seq = realise(C, regs, opframe(O), k, tl) + if seq is None: return None + rem.add(i) + for n_, op in enumerate(seq): add.append((k + n_ * 1e-7, op)) + return rem, add diff --git a/research/p104-findings-nov15/code/relay/pin.py b/research/p104-findings-nov15/code/relay/pin.py new file mode 100644 index 0000000000..b2366f8e3a --- /dev/null +++ b/research/p104-findings-nov15/code/relay/pin.py @@ -0,0 +1,97 @@ +"""BIG-RELAY pin.py: per transfer j->i at a relay frame F (ADD i += c j), the shear window (t_lo, t_hi) bounded by the +nearest blocking ops (write into j from a non-i source, read of i by a non-j target; F-ops excluded; SCALEs absorbable). +pinned = i and j co-reside in the window ONLY at F. courier = some third register D meets j (after t_lo) and then i (before t_hi) +at frames != F, in time order, with no new stop.""" +import collections, pickle, bisect, math +from lib import * +W = load(); NP, mains, fixed, dim = setup(W); P = W['program'] +X = pickle.load(open('explore.pkl', 'rb')) +NO = W['owned_fields'] +byreg = collections.defaultdict(list) +for t, e in enumerate(P): + if e[0] in ('ADD', 'COPY'): byreg[e[1]].append(t); byreg[e[2]].append(t) + elif e[0] != 'MOVE': byreg[e[1]].append(t) +# stays: register -> list of (t_from, t_to, frame) +cur = list(W['initial']) + [None] * NO; start = [0] * (NP + NO); ST = collections.defaultdict(list) +for t, e in enumerate(P): + if e[0] == 'MOVE': ST[e[1]].append((start[e[1]], t, cur[e[1]])); cur[e[1]] = e[3]; start[e[1]] = t + elif e[0] == 'INIT': cur[e[1]] = e[2]; start[e[1]] = t + elif e[0] == 'CLEAR': ST[e[1]].append((start[e[1]], t, cur[e[1]])); cur[e[1]] = None +for r in range(NP): ST[r].append((start[r], len(P), cur[r])) +byframe = collections.defaultdict(list) # frame -> stays (r, a, b) +for r, L in ST.items(): + for a, b, G in L: + if G is not None: byframe[G].append((r, a, b)) +def meets(r, lo, hi, F): + """{(other, frame, a, b)} co-residence intervals of r with others inside (lo,hi), frame != F.""" + out = [] + for a, b, G in ST[r]: + if G == F or b <= lo or a >= hi: continue + a2, b2 = max(a, lo), min(b, hi) + for (o, c, d) in byframe[G]: + if o != r and c < b2 and d > a2: out.append((o, G, max(a2, c), min(b2, d))) + return out +HALF = {}; NP_ = {}; rows = []; pinned_frames = 0; stat = collections.Counter(); blk = collections.Counter() +for F in X['relay']: + hs = {s[0] for s in X['stops'][F]}; Fpin = False; Fcour = True; Fmh = []; Fblk = [] + for t, e in enumerate(P): + pass + for t in sorted({t for h in hs for t in byreg[h] if evframe(P[t]) == F}): + e = P[t] + if e[0] != 'ADD': continue + i, j = e[1], e[2] + def isblk(u): + o = P[u] + if evframe(o) == F or o[0] != 'ADD': return False + return (o[1] == j and o[2] != i) or (o[2] == i and o[1] != j) + before = [u for u in byreg[i] + byreg[j] if u < t and isblk(u)]; after = [u for u in byreg[i] + byreg[j] if u > t and isblk(u)] + lo = max(before) if before else -1; hi = min(after) if after else len(P) + for u in (max(before) if before else None, min(after) if after else None): + if u is None: continue + o = P[u]; who = o[2] if o[1] in (i, j) else o[1] + blk[('write-into-j' if o[1] == j else 'read-of-i', 'fixed' if fixed(who) else 'helper', dim[o[4]])] += 1 + co = [m for m in meets(i, lo, hi, F) if m[0] == j] + if co: stat['transfer not pinned'] += 1; continue + stat['pinned'] += 1; Fpin = True + # courier: D meets j in (lo, t) then i in (that time, hi) + mj = {} + for (o, G, a, b) in meets(j, lo, hi, F): + if o not in hs or True: mj.setdefault(o, []).append(a) + ok = False + for (o, G, a, b) in meets(i, lo, hi, F): + if o in mj and min(mj[o]) < b and not fixed(o): ok = True; break + stat['pinned, courier exists' if ok else 'pinned, no courier'] += 1 + # multi-hop: earliest-arrival flood from j (time lo) through helpers, reach i before hi + import heapq + arr = {j: lo}; pq = [(lo, j)]; reach = False; nst = 0 + while pq and nst < 4000: + a0, r = heapq.heappop(pq); nst += 1 + if a0 > arr.get(r, 1e18): continue + for (o, G, a, b) in meets(r, a0, hi, F): + if fixed(o): continue + ta = max(a, a0) + if o == i: reach = True; break + if ta < arr.get(o, 1e18): arr[o] = ta; heapq.heappush(pq, (ta, o)) + if reach: break + stat['pinned, multi-hop path' if reach else 'pinned, no multi-hop path'] += 1 + Fmh.append(reach) + if not reach: + A = {o for (o, G, a, b) in meets(j, lo, hi, F) if not fixed(o)}; B = {o for (o, G, a, b) in meets(i, lo, hi, F) if not fixed(o)} + HALF.setdefault(F, {})[(i, j)] = bool(A or B) + if not reach: NP_.setdefault(F, set()).add((i, j)) + bl = [P[u] for u in ((max(before),) if before else ()) + ((min(after),) if after else ())] + Fblk.append(all(fixed(o[2] if o[1] in (i, j) else o[1]) for o in bl)) + if not ok: Fcour = False + rows.append((F, Fpin, Fcour, all(Fmh), any(Fblk))) +print(stat); print('frames with >=1 pinned transfer', sum(1 for r in rows if r[1]), 'of', len(rows), '; all pinned transfers courier-able', sum(1 for r in rows if r[1] and r[2])) +print('frames whose pinned transfers all have a multi-hop helper path', sum(1 for r in rows if r[3]), '; frames with a pinned transfer blocked on both sides by FIXED-register ops', sum(1 for r in rows if r[4])) +print('blockers (kind, partner, dim):', blk.most_common(12)) +pickle.dump(dict(rows=rows, nopath=NP_, half=HALF), open('pin.pkl', 'wb')) +gain = {r[0]: r[6] for r in X['rows']} +L = 2 * math.log(2) +c1 = sum(max(0, gain[F] - L) for F in NP_); +mt = {F: max(len({p[0] for p in v}), len({p[1] for p in v})) for F, v in NP_.items()} +c2 = sum(max(0, gain[F] - L * mt[F]) for F in NP_) +print('gain all relay frames %.1f; ceiling m_F=1 %.1f; ceiling m_F=max(#targets,#sources) %.1f' % (sum(gain.values()), c1, c2)) +print('m_F dist', collections.Counter(mt.values()), 'frames with gain_F > L*m_F', sum(1 for F in NP_ if gain[F] > L * mt[F] + 1e-9)) +import collections as C2; print('no-path pairs per frame', C2.Counter(len(v) for v in NP_.values())) diff --git a/research/p104-findings-nov15/code/relay/rehost.py b/research/p104-findings-nov15/code/relay/rehost.py new file mode 100644 index 0000000000..977d62f41f --- /dev/null +++ b/research/p104-findings-nov15/code/relay/rehost.py @@ -0,0 +1,30 @@ +"""BIG-RELAY rehost.py: re-host ALL events of relay frame F at one other frame G (joint relabel, timing unchanged). +Each stopping helper h (a_h -> F -> b_h) must be able to sit at G during its F-stay: a_h < G < b_h (new stop), or G == a_h / G == b_h +(merge into the neighbouring stop: the MOVE is shifted, legal only if h has no op in between, true here since the stay at F holds only F-ops). +Exact hop-cost delta (D0/call); rows unchanged by construction (same ops, same order).""" +import pickle, collections +from lib import * +W = load(); NP, mains, fixed, dim = setup(W) +X = pickle.load(open('explore.pkl', 'rb')) +def span(b): + s = {0} + for v in b: s |= {x ^ v for x in s} + return frozenset(s) +SP = [span(b) for b in W['frames']] +best = [] +for F in X['relay']: + S = X['stops'][F]; cands = [] + for G in range(len(SP)): + if G == F: continue + d = 0.0; ok = True + for (h, ar, lv, a, b) in S: + old = -f(dim[F] - dim[a]) - f(dim[b] - dim[F]) + if G == a or G == b: new = -f(dim[b] - dim[a]) + elif SP[a] < SP[G] < SP[b]: new = -f(dim[G] - dim[a]) - f(dim[b] - dim[G]) + else: ok = False; break + d += new - old + if ok: cands.append((d, G)) + if cands: best.append((min(cands), F, len(cands))) +print('relay frames with any joint re-host target', len(best), 'of', len(X['relay'])) +print('improving (delta<0):', [(F, G, round(d, 4)) for (d, G), F, n in best if d < -1e-9][:20]) +print('delta dist', collections.Counter(round(d, 3) for (d, G), F, n in best)) diff --git a/research/p104-findings-nov15/code/relay/rehost2.py b/research/p104-findings-nov15/code/relay/rehost2.py new file mode 100644 index 0000000000..2be5791545 --- /dev/null +++ b/research/p104-findings-nov15/code/relay/rehost2.py @@ -0,0 +1,34 @@ +"""BIG-RELAY rehost2.py: joint relabel of F -> G for EVERY register whose chain visits F (interior stops, births at F, ends at F). +New chain: replace F by G; drop duplicates if G equals a neighbour; must stay strictly ascending; mains never visit relay frames. +Cost = exact helper cost f(d_end-d_start) - sum f(gaps) (D0/call), delta vs base.""" +import pickle, collections, sys +from lib import * +W = load(); NP, mains, fixed, dim = setup(W); ch = chains(W) +X = pickle.load(open('explore.pkl', 'rb')) +def span(b): + s = {0} + for v in b: s |= {x ^ v for x in s} + return frozenset(s) +SP = [span(b) for b in W['frames']] +visit = collections.defaultdict(list) +for r in range(NP): + for k, (fr, t) in enumerate(ch[r]): visit[fr].append(r) +res = [] +for F in X['relay']: + regs = sorted(set(visit[F])); assert not any(r in mains for r in regs) + base = sum(hcost([dim[x] for x, _ in ch[r]]) for r in regs) + best = None + for G in range(len(SP)): + if G == F: continue + tot = 0.0; ok = True + for r in regs: + c = [x for x, _ in ch[r]]; c = [G if x == F else x for x in c] + c2 = [c[0]] + [c[i] for i in range(1, len(c)) if c[i] != c[i - 1]] + if any(not SP[c2[i]] < SP[c2[i + 1]] for i in range(len(c2) - 1)) or len(c2) < 2: ok = False; break + tot += hcost([dim[x] for x in c2]) + if ok and (best is None or tot - base < best[0]): best = (tot - base, G) + res.append((F, len(regs), best)) +ok = [r for r in res if r[2] is not None] +print('relay frames with a legal joint re-host', len(ok), '; improving', [(F, n, G, round(d, 4)) for F, n, (d, G) in ok if d < -1e-9]) +print('nonimproving deltas', sorted(round(r[2][0], 3) for r in ok if r[2][0] >= -1e-9)) +pickle.dump(res, open('rehost2.pkl', 'wb')) diff --git a/research/p104-findings-nov15/code/relay/runall.py b/research/p104-findings-nov15/code/relay/runall.py new file mode 100644 index 0000000000..94d38e5e00 --- /dev/null +++ b/research/p104-findings-nov15/code/relay/runall.py @@ -0,0 +1,11 @@ +import collections, pickle +from drive import * +base_ops = [(float(i), e) for i, e in enumerate(W['program'])] +tl = timeline(base_ops, NP, W['owned_fields'], W['initial']) +res = {}; t0 = time.time() +for F in X['relay']: + r, why = remove_frame(list(base_ops), F, X['stops'][F], tl) + res[F] = (why, gainF(F)) +c = collections.Counter(v[0].split()[0] for v in res.values()); print(c, '%.0fs' % (time.time() - t0)) +ok = [F for F, v in res.items() if v[0] == 'ok']; print('ok frames', ok, 'gain', sum(res[F][1] for F in ok)) +pickle.dump(res, open('runall.pkl', 'wb')) diff --git a/research/p104-findings-nov15/code/relay/shape.py b/research/p104-findings-nov15/code/relay/shape.py new file mode 100644 index 0000000000..fda84da3a1 --- /dev/null +++ b/research/p104-findings-nov15/code/relay/shape.py @@ -0,0 +1,63 @@ +"""BIG-RELAY shape.py: what each relay frame does. Per frame: registers touching it, F-op transfer graph (j->i for i += c j), +mutual pairs (butterflies), and for pinned no-path pairs whether F is the JOIN of the two predecessor frames and/or the MEET of +the two successor frames (the 'diamond' [a_i+a_j, b_i^b_j] collapses to F).""" +import pickle, collections +from lib import * +W = load(); NP, mains, fixed, dim = setup(W); P = W['program']; ch = chains(W) +X = pickle.load(open('explore.pkl', 'rb')); PN = pickle.load(open('pin.pkl', 'rb'))['nopath'] +def span(b): + s = {0} + for v in b: s |= {x ^ v for x in s} + return frozenset(s) +SP = [span(b) for b in W['frames']]; idx = {s: i for i, s in enumerate(SP)} +def join(A, B): + s = set(A) + for v in B: s |= {x ^ v for x in s} + return frozenset(s) +ops = collections.defaultdict(list) +for t, e in enumerate(P): + if e[0] == 'ADD' and e[4] in PN: ops[e[4]].append(e) +def nb(r, F): + c = [x for x, _ in ch[r]]; k = c.index(F); return (c[k - 1] if k else None), (c[k + 1] if k + 1 < len(c) else None) +st = collections.Counter(); shp = collections.Counter(); dia = collections.Counter() +for F in PN: + E = ops[F]; tr = {(e[1], e[2]) for e in E} + mutual = sum(1 for (i, j) in tr if (j, i) in tr) // 2 + regs = {x for e in E for x in e[1:3]} + shp[('regs', min(len(regs), 9), 'transfers', min(len(tr), 12), 'mutual', mutual)] += 1 + for (i, j) in PN[F]: + ai, bi = nb(i, F); aj, bj = nb(j, F) + if None in (ai, aj, bi, bj): dia['endpoint at F'] += 1; continue + J = join(SP[ai], SP[aj]); M = SP[bi] & SP[bj] + dia[('F==join(a_i,a_j)', J == SP[F], 'F==meet(b_i,b_j)', M == SP[F])] += 1 +print('transfer-graph shapes (top):'); [print(' ', k, v) for k, v in shp.most_common(10)] +print('pinned no-path pairs, diamond test:', dict(dia)) +# ceiling: each diamond-collapsed pinned pair needs a courier that visits a_j (after j's pick) and b_i (before i's read), which no +# helper does today (flood): >= 1 new stop (>= f(2)-2f(1) = 2 ln 2) per distinct (a_j, b_i). +import math +L = 2 * math.log(2); gain = {r[0]: r[6] for r in X['rows']} +need = collections.Counter(); +for F in PN: + s = set() + for (i, j) in PN[F]: + ai, bi = nb(i, F); aj, bj = nb(j, F) + if None in (ai, aj, bi, bj): continue + if join(SP[ai], SP[aj]) == SP[F] and (SP[bi] & SP[bj]) == SP[F]: s.add((aj, bi)) + need[F] = len(s) +cz = sum(max(0, gain[F] - L * need[F]) for F in PN); call = sum(gain.values()) +print('relay gain total %.1f D0/call; frames with >=1 collapsed pinned pair %d; ceiling after courier stops %.1f D0/call; frames still positive %d' + % (call, sum(1 for F in PN if need[F]), cz, sum(1 for F in PN if gain[F] > L * need[F] + 1e-9))) +print('need dist', sorted(collections.Counter(need.values()).items())) +H = pickle.load(open('pin.pkl', 'rb'))['half'] +need2 = {} +for F in PN: + best = {} + for (i, j) in PN[F]: + ai, bi = nb(i, F); aj, bj = nb(j, F) + if None in (ai, aj, bi, bj): continue + if join(SP[ai], SP[aj]) == SP[F] and (SP[bi] & SP[bj]) == SP[F]: + k = (aj, bi); c = 1 if H[F][(i, j)] else 2; best[k] = min(best.get(k, 9), c) + need2[F] = sum(best.values()) +cz2 = sum(max(0, gain[F] - L * need2[F]) for F in PN) +print('refined (2 new stops when no helper touches either end): ceiling %.1f D0/call, frames positive %d' % (cz2, sum(1 for F in PN if gain[F] > L * need2[F] + 1e-9))) +print('half-touched pairs', sum(1 for F in H for v in H[F].values() if v), 'of', sum(len(H[F]) for F in H)) diff --git a/research/p104-findings-nov15/code/relay/show.py b/research/p104-findings-nov15/code/relay/show.py new file mode 100644 index 0000000000..68fc12cb63 --- /dev/null +++ b/research/p104-findings-nov15/code/relay/show.py @@ -0,0 +1,10 @@ +import sys, pickle +from lib import * +W = load(); NP, mains, fixed, dim = setup(W); P = W['program']; ch = chains(W) +X = pickle.load(open('explore.pkl', 'rb')) +for F in map(int, sys.argv[1:]): + S = X['stops'][F]; print('=== F', F, 'dim', dim[F], W['frames'][F]) + for s in S: + h = s[0]; print(' helper', h, 'chain dims', [dim[c[0]] for c in ch[h]], 'frames', [c[0] for c in ch[h]], 'arr', s[1], 'leave', s[2]) + for i, e in enumerate(P): + if evframe(e) == F: print(' op', i, e) diff --git a/research/p104-findings-nov15/code/slack/icap.py b/research/p104-findings-nov15/code/slack/icap.py new file mode 100644 index 0000000000..f4d819aaf4 --- /dev/null +++ b/research/p104-findings-nov15/code/slack/icap.py @@ -0,0 +1,42 @@ +"""SLACK icap.py [C-param]: reprice layouts with INTERVAL line caps. A line covering stages a..b is one idle op scaling one register by +D_a...D_b; by SL its end-referred vectors lie in R^{Z_a cap ... cap Z_b} and are orthogonal to Y_a..Y_b (rows) / X_a..X_b (cols), and +lines sharing a stage are independent. Hence N[a,b] := #lines covering [a,b] <= |Z_[a,b]| - rank(Y_a..Y_b | Z_[a,b]). +layout_price / step1 impose this ONLY for b = a+1 (per-boundary 'cap |S|-4'); here every interval is imposed with rank r (r=4 = Crossing +rank; r=3 = worst case if the Crossing Lemma fails on the restricted set). Greedy: at each boundary keep the OLDEST lines subject to all +interval caps (same rule that reproduces built B).""" +import sys; sys.path.insert(0, '.') +from layout_price import price, two_arm +from collections import Counter +L = 17; nl = 8 +def lines_iv(masks, r_of): + Z = [frozenset(i for i, ms in enumerate(masks) if k in ms) for k in range(L)] + m = [len(z) - 2 for z in Z] + alive = []; out = Counter() + for k in range(L): + if k > 0: + if k - 1 == nl - 1: keep_list = [] + else: + keep_list = [] + for s in alive: # oldest first + trial = keep_list + [s] + ok = len(trial) <= m[k] + for a in range(min(trial), k): # interval [a, k] + Za = frozenset.intersection(*Z[a:k + 1]); cnt = sum(1 for t in trial if t <= a) + if cnt > max(len(Za) - r_of(a, k), 0): ok = False; break + if ok: keep_list.append(s) + kept = Counter(keep_list); rem = Counter(alive) - kept + for s, v in rem.items(): out[k - s] += v + alive = sorted(keep_list) + alive += [k] * (m[k] - len(alive)) + for s in alive: out[L - s] += 1 + return out, [len(z) for z in Z] +def invoice_iv(masks, r_of): + T = Counter(8 * len(ms) + 1 for ms in masks) + lc, Z = lines_iv(masks, r_of) + P = Counter({8 * l: v for l, v in lc.items()}) + thin = len(masks) + 2 - 2 * L; assert thin >= 0 + for _ in range(thin): + rr = min((x for x in P if x % 8 == 0 and P[x] > 0), default=None) + if rr is None: P[1] += 1; continue + P[rr] -= 1; P[rr + 1] += 1 + return T, +P, Z diff --git a/research/p104-findings-nov15/code/slack/icap_check.py b/research/p104-findings-nov15/code/slack/icap_check.py new file mode 100644 index 0000000000..e18a791898 --- /dev/null +++ b/research/p104-findings-nov15/code/slack/icap_check.py @@ -0,0 +1,43 @@ +"""[C-param] SLACK icap_check.py: the interval-capped line pricing correction (SLACK PROOF A), on a pinned handful of layouts. +Old pricer: caps only two-stage lines (per-boundary |S|-4). Corrected: every interval I gets N(I) <= |Z_I| - 4 (r = 4, the Crossing rank; +Extended Crossing Lemma, PROOF B/E). All prices are min over the three native columns (twins deleted / shadows / shadows+2 relays) of kappa/kappa_B. +Asserts, with the numbers quoted in README section 3: + - the nested B layout is unchanged by the correction (1.0950 / 1.0031 / 0.9773 both ways) [control]; + - 'B + one lone extra role' prices > +0.15% (+0.20 to +0.59) with the old boundary caps and <= 0 with interval caps (all four layouts tested); + - pair k=0 d=4 + slack 1..4: +1.56% old -> +0.20% corrected (r=4 only on intervals where the Crossing Lemma applies). +Prints PASS_ICAP.""" +import sys, os +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from icap import * +from layout_price import invoice +def capS(ms): + cap = [] + for k in range(L - 1): + if k == nl - 1: cap.append(0); continue + S = sum(1 for m in ms if k in m and k + 1 in m); cap.append(max(S - 4, 0)) + return cap +def dying(c, k, d): + ms = two_arm(); full = set(range(k, nl)) + for _ in range(c): ms.remove(full); ms.append(set(range(k, d + 1))) + return ms +def mk_r(ms, rfail): + Z = [frozenset(i for i, m in enumerate(ms) if k in m) for k in range(L)] + def r(a, b): + ZI = frozenset.intersection(*Z[a:b + 1]) + if any(Z[k] <= ZI or Z[k + 1] <= ZI for k in range(a, b)): return 4 + return rfail + return r +def old(ms): T, P, Z = invoice(ms, L, capS(ms)); return price(T, P, '', quiet=True) +def new(ms): T, P, Z = invoice_iv(ms, mk_r(ms, 4)); return price(T, P, '', quiet=True) +B_old = old(two_arm()); B_new = new(two_arm()) +print('B control old', ['%.4f' % x for x in B_old], 'interval', ['%.4f' % x for x in B_new]) +assert ['%.4f' % x for x in B_old] == ['%.4f' % x for x in B_new] == ['1.0950', '1.0031', '0.9773'] +g = lambda x: 100 * (min(p / q for p, q in zip(x, B_old)) - 1) +for je, ee in ((1, 3), (0, 4), (2, 5), (3, 6)): + ms = two_arm(); ms.append(set(range(je, ee + 1))) + a, b = g(old(ms)), g(new(ms)); print('B + lone %d..%d old %+.2f interval %+.2f' % (je, ee, a, b)) + assert a > 0.15 and b <= 0.0 +ms = dying(2, 0, 4); ms.append(set(range(1, 5))) +a, b = g(old(ms)), g(new(ms)); print('pair k=0 d=4 + slack 1..4 old %+.2f interval %+.2f' % (a, b)) +assert abs(a - 1.56) < 0.006 and abs(b - 0.20) < 0.006 +print('PASS_ICAP') diff --git a/research/p104-findings-nov15/code/slack/layout_price.py b/research/p104-findings-nov15/code/slack/layout_price.py new file mode 100644 index 0000000000..3fb9a03d64 --- /dev/null +++ b/research/p104-findings-nov15/code/slack/layout_price.py @@ -0,0 +1,88 @@ +"""CALLEROPEN layout_price.py [C-param, PRICEKIT]: price a rho=2 mask layout from (i) role masks (stage sets), (ii) per-boundary +line caps. Ledger [P]: T = sum_roles T[8 deg+1] (odd phase), idle incidences at stage k = |Z_k| - 2 (Lemma Det), thin = n+2-2L +QUERY P[1]; lines = intervals of consecutive stages, #lines at k = |Z_k|-2, #crossing k|k+1 <= cap_k. Line lengths: at each +boundary continue the OLDEST lines up to min(m_k, m_k+1, cap_k) (most unequal arrangement for convex f) -> P[8 len]. +Natives: FABLE5 allb_pk set (twins deleted / shadows / shadows+2 relays) for 104 ports, 17 calls.""" +import sys, copy, math +import os; _HERE = os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0, os.path.join(_HERE, '..')); import pricekit as pk +from collections import Counter +W = pk.load(os.path.join(_HERE, '..', '..', 'data', 'w7546.json.gz')); C7 = pk.load(os.path.join(_HERE, '..', '..', 'data', '7113_CALLER.json.gz')) +n0 = pk.native_from_word(W); base = pk.price(pk.invoice(pk.caller_invoice(C7), n0), 30); K = base['kappa']['lo'] +W2 = copy.deepcopy(W); W2['hosted'] = {}; nt = pk.native_from_word(W2) +nm = dict(nt); P_ = Counter(n0['P']) +for c in ('X2', 'Y2'): P_.subtract(n0['main_moves'][c]) +nm['P'] = +P_; nm['T'] = Counter(n0['T']); nm['source'] = 'param' +nr = dict(nt); nr['T'] = Counter(nt['T']); nr['P'] = Counter(nt['P']); nr['source'] = 'param' +nr['T'][9] += 2 * 96; nr['P'][1] += 96; nr['P'][8] += 2 * 96; nr['P'][1] += 96 +NAT = (('twins deleted', nm), ('shadows', nt), ('shadows+2 relays', nr)) +f = lambda r: r * math.log(r) if r > 1 else 0.0 +def lines(m, cap): + """m[k] lines needed at stage k (k=0..L-1); cap[k] max crossing k|k+1. Returns Counter of line lengths.""" + alive = [] # start stages, oldest first + out = Counter() + for k in range(len(m)): + if k > 0: + keep = min(m[k - 1], m[k], cap[k - 1], len(alive)) + for s in alive[keep:]: out[k - s] += 1 + alive = alive[:keep] + alive += [k] * (m[k] - len(alive)) + for s in alive: out[len(m) - s] += 1 + return out +def invoice(masks, L, cap, cov=2): + """masks: list of role stage-sets; cap: list of L-1 caps.""" + Z = [sum(1 for ms in masks if k in ms) for k in range(L)] + m = [z - cov for z in Z]; assert min(m) >= 0, Z + T = Counter(8 * len(ms) + 1 for ms in masks) + P = Counter({8 * l: v for l, v in lines(m, cap).items()}) + n = len(masks); thin = n + 2 - 2 * L; assert thin >= 0, (n, L) + # thin QUERYs absorbed as IDLE_ODD into the shortest lines (as the built B does: 4 x r9) + for _ in range(thin): + r = min((r for r in P if r % 8 == 0 and P[r] > 0), default=None) + if r is None: P[1] += 1; continue + P[r] -= 1; P[r + 1] += 1 + return T, +P, Z +def price(T, P, lab, nJ=17, quiet=False): + Ta = Counter({r: v * 104 for r, v in T.items()}); Pa = Counter({r: v * 104 for r, v in P.items()}) + res = [] + for nn, nat in NAT: + pr = pk.price(pk.invoice(dict(T=Ta, P=Pa, nJ=nJ), nat), 30); k = pr['kappa'] + res.append(float(k['lo'] / K) if isinstance(k, dict) and k.get('lo') else 0.0) + D = sum(v * f(r) for r, v in T.items()) - sum(v * f(r) for r, v in P.items()) + if not quiet: + print('%-58s S/port %4d D0/port %7.1f header %3d | td %.4f sh %.4f rel %.4f' % (lab, pk.S_of(T, P), D, sum((r - 1) // 8 * v for r, v in T.items()), *res)) + return res +def two_arm(L=17, nl=8, hubs=2, grow=2, extra=0, deg1=True): + """B-type: hubs on all stages; left arm stages 0..nl-1 (centre nl-1), right nl..L-1 (centre nl). At left stage k (0-based) + 'grow' own roles enter (k>=1 or always), mask k..nl-1; 'extra' own roles enter at stage 0 additionally.""" + masks = [set(range(L)) for _ in range(hubs)] + for k in range(nl): + cnt = grow + (extra if k == 0 else 0) + masks += [set(range(k, nl)) for _ in range(cnt)] + for k in range(L - 1, nl - 1, -1): + cnt = grow + (extra if k == L - 1 else 0) + masks += [set(range(nl, k + 1)) for _ in range(cnt)] + return masks +def caps_B(Z, L, nl, bonus=0): + """interior caps |S|-4 ; centre boundary: S = hubs only -> crossing <= hubs-? use |S|-4 floored at 0; outer boundaries |S|-4+bonus.""" + cap = [] + for k in range(L - 1): + S = Z[k] if k < nl - 1 else (Z[k + 1] if k >= nl else None) + if S is None: cap.append(0); continue # centre (thin block between): no line crosses (S = hubs, covered) + c = S - 4 + if k == 0 or k == L - 2: c += bonus + cap.append(max(c, 0)) + return cap +if __name__ == '__main__': + print(pk.cite('control #7546 on #7113', base)) + Zof = lambda ms: [sum(1 for m in ms if k in m) for k in range(17)] + ms = two_arm(); T, P, Z = invoice(ms, 17, caps_B(Zof(ms), 17, 8)) + print('model B: Z', Z, 'P', sorted(P.items())) + price(T, P, 'model B (built B: 1.0950/1.0031/0.9773)') + T, P, Z = invoice(ms, 17, caps_B(Zof(ms), 17, 8, bonus=2)); price(T, P, 'B + outer bonus (2 lines cross each outer boundary)') + T, P, Z = invoice(ms, 17, caps_B(Zof(ms), 17, 8, bonus=1)); price(T, P, 'B + outer bonus 1') + ms2 = two_arm(extra=-2); Z2 = Zof(ms2); print('tight Z', Z2, 'n', len(ms2)) + T, P, Z = invoice(ms2, 17, caps_B(Z2, 17, 8)); price(T, P, 'tight two-hub |Z_k|=2k (no own role at outer stages)') + for nl in (7, 9): + ms3 = two_arm(nl=nl); T, P, Z = invoice(ms3, 17, caps_B(Zof(ms3), 17, nl)); price(T, P, 'B with centre at %d|%d' % (nl, nl + 1)) + # uncapped ceiling (all lines run full) for reference + T, P, Z = invoice(ms, 17, [99] * 16); price(T, P, 'B masks, NO caps (unreachable reference)') diff --git a/research/p104-findings-nov15/code/slack/rs3.py b/research/p104-findings-nov15/code/slack/rs3.py new file mode 100644 index 0000000000..880e3e97de --- /dev/null +++ b/research/p104-findings-nov15/code/slack/rs3.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python3 +"""SLACK rs3.py = CALLEROPEN rs2.py with INTERVAL lines (stage tuple, count, support). +CALLEROPEN rs2.py [N]: same model as rootsearch.py (units, product -I, HOST pair invariant, supports in masks, PREFIX +COMMUTATION before/after every stage, optional crossing idle lines) with an ANALYTIC Jacobian. Solver scipy least_squares +(trf), fixed number of random starts per row (stopping rule). solved = max|residual| < 1e-10.""" +import sys, json +import numpy as np +from scipy.optimize import least_squares +class Model: + def __init__(s, n, order, t, lines=()): + s.n = n; units = []; stages = [] + for it in order: + if it[0] == 'S': stages.append((len(units), list(it[1]))); units += [list(it[1]), list(it[1])] + else: units.append(list(range(n)) if len(it) == 1 else list(it[1])) + s.units = units; s.stages = stages; s.N = N = len(units) + s.idx = []; off = 0 + for sup in units: s.idx.append((off, np.array(sup))); off += 2 * len(sup) + s.lidx = [] + for spec in lines: # (stage tuple, count, support or None = intersection of the stage masks) + st_, c, S = spec + if S is None: S = sorted(set.intersection(*[set(stages[a][1]) for a in st_])) + for _ in range(c): s.lidx.append((tuple(st_), None, off, np.array(sorted(S)))); off += 2 * len(S) + s.dim = off; s.tk = t if isinstance(t, (list, tuple)) else [t] * len(stages) + def unpack(s, z): + X = np.zeros((s.N, s.n)); Y = np.zeros((s.N, s.n)) + for i, (o, sup) in enumerate(s.idx): + k = len(sup); X[i, sup] = z[o:o + k]; Y[i, sup] = z[o + k:o + 2 * k] + return X, Y + def fj(s, z): + n, N = s.n, s.N; X, Y = s.unpack(z); I = np.eye(n) + T = [I - 2 * np.outer(X[i], Y[i]) for i in range(N)] + pref = [I] + for i in range(N): pref.append(T[i] @ pref[-1]) + rows = []; jac = [] + def blank(m): return np.zeros((m, s.dim)) + # 1 isotropy + r = np.sum(X * Y, axis=1); J = blank(N) + for i, (o, sup) in enumerate(s.idx): + k = len(sup); J[i, o:o + k] = Y[i, sup]; J[i, o + k:o + 2 * k] = X[i, sup] + rows.append(r); jac.append(J) + # product and prefixes: d(A_{u}) where A_u = T_{u-1}..T_0 ; dA_u/dunit i (i=1 (pair entry): r=2 [unproved]; + (ii) slack dying at K with je<=a<=d, je>k: r=3 [G: drop <= 1, case t(v)=a open]; (iii) slack entering <= k: r=2 [not analysed].""" +import sys, io, contextlib; sys.path.insert(0,'.') +with contextlib.redirect_stdout(io.StringIO()): + import step1 +from icap import * +from layout_price import invoice +def Zs(ms): return [frozenset(i for i,m in enumerate(ms) if k in m) for k in range(L)] +def mkr(ms,k,d,je,eend,mode): + Z=Zs(ms); K=d+1 + def r(a,b): + ZI=frozenset.intersection(*Z[a:b+1]) + if any(Z[q]<=ZI or Z[q+1]<=ZI for q in range(a,b)): return 4 + if not (a<=d and b>=K): return 4 if mode=='opt' else 2 # no death crossed but criterion fails: not expected + if a==k and k>=1: return 2 if mode!='opt' else 4 + if je<=k: return 2 if mode!='opt' else 4 + if eend==d and je<=a<=d: return 3 if mode!='opt' else 4 + return 4 + return r +B=None +def pr(ms,r): + T,P,Z=invoice_iv(ms,r); return price(T,P,'',quiet=True) +B=pr(two_arm(),lambda a,b:4) +g=lambda x:100*(min(p/q for p,q in zip(x,B))-1) +def U(ms,j): return sum(1 for m in ms if min(m)<=j) +left=lambda ms:[m for m in ms if max(m)= +0.5%%: %d of %d'%(rows[0][1], sum(1 for t in rows if t[1]>=0.5), len(rows))) +k0=[t for t in rows if ' k=0 ' in t[0]] +print('k=0 only: max proved %+.2f (%s); >=0.5: %d of %d'%(k0[0][1],k0[0][0],sum(1 for t in k0 if t[1]>=0.5),len(k0))) +for t in k0[:5]: print(' %-28s %+.2f | %+.2f'%t) diff --git a/research/p104-findings-nov15/code/slack/run5.py b/research/p104-findings-nov15/code/slack/run5.py new file mode 100644 index 0000000000..588c6153ce --- /dev/null +++ b/research/p104-findings-nov15/code/slack/run5.py @@ -0,0 +1,26 @@ +"""SLACK run5 [C-param]: k>=1 survivors priced with rank 2 ONLY on intervals starting at the pair entry a=k (where rank 2 is EXACT, PROOF H), +rank 4 elsewhere; vs k=0 survivors with rank 3 only in the open case (ii).""" +import sys, io, contextlib; sys.path.insert(0,'.') +with contextlib.redirect_stdout(io.StringIO()): + import step1 +from icap import * +exec(open('run4.py').read().split('rows=[]')[0].split('import step1')[1].replace('from icap import *','').replace('from layout_price import invoice','')) +def mk(ms,k,d): + Z=Zs(ms); K=d+1 + def r(a,b): + ZI=frozenset.intersection(*Z[a:b+1]) + if any(Z[q]<=ZI or Z[q+1]<=ZI for q in range(a,b)): return 4 + return 2 if (a==k and b>=K) else 4 + return r +out=[] +for c in (2,1): + for d in range(1,nl-2): + for je in range(0,d+1): + for eend in (d,nl-1): + ms=step1.dying([('L',1,d,c,0)]); ms.append(set(range(je,eend+1))) + if killed(ms,c,1,d): continue + out.append((g(pr(ms,mk(ms,1,d))),'c=%d k=1 d=%d slack %d..%d'%(c,d,je,eend))) + for d in range(2,nl-2): + ms=step1.dying([('L',1,d,c,0)]); out.append((g(pr(ms,mk(ms,1,d))),'c=%d k=1 d=%d NO slack (DYINGPAIR-dead)'%(c,d))) +out.sort(reverse=True) +for t in out[:10]: print(' %+.2f %s'%t) diff --git a/research/p104-findings-nov15/code/slack/run6.py b/research/p104-findings-nov15/code/slack/run6.py new file mode 100644 index 0000000000..b8789a8933 --- /dev/null +++ b/research/p104-findings-nov15/code/slack/run6.py @@ -0,0 +1,46 @@ +"""SLACK run6 [P-caps + C-param]: CONSERVATIVE proved interval ranks for the whole pair/single + slack menu. + r(I) = 4 if Lemma B criterion; else 4 - min(2, u), u = #roles of Omega_a = U_a \\ Z_I NOT certified touched by X_{= 1, + |U_{t-1}| <= 2t+2 for k' < t <= d (top index on v in V cap R^Delta; recursion at k').""" +import sys, io, contextlib; sys.path.insert(0,'.') +with contextlib.redirect_stdout(io.StringIO()): + import step1 +from icap import * +def Zs(ms): return [frozenset(i for i,m in enumerate(ms) if k in m) for k in range(L)] +def pr(ms,r): + T,P,Z=invoice_iv(ms,r); return price(T,P,'',quiet=True) +B=pr(two_arm(),lambda a,b:4); g=lambda x:100*(min(p/q for p,q in zip(x,B))-1) +def U(ms,j): return sum(1 for m in ms if min(m)<=j) +left=lambda ms:[m for m in ms if max(m)=1 and all(U(ms,t-1)<=2*t+2 for t in range(min(ents)+1,d+1)) + return old or new +def mkr(ms,k,d,c): + Z=Zs(ms); idx=list(range(len(ms))) + cert=set() + if U(ms,k)==2*k+4: cert={i for i in idx if min(ms[i])==k and max(ms[i])==d and len(ms[i])==d-k+1} # dying set + def r(a,b): + ZI=frozenset.intersection(*Z[a:b+1]) + if any(Z[q]<=ZI or Z[q+1]<=ZI for q in range(a,b)): return 4 + Om=[i for i in idx if min(ms[i])<=a and i not in ZI] + u=sum(1 for i in Om if not (i in cert and k= +0.5%%: %d'%sum(1 for t in rows if t[0]>=0.5)) diff --git a/research/p104-findings-nov15/code/slack/sol_ctl_2.npy b/research/p104-findings-nov15/code/slack/sol_ctl_2.npy new file mode 100644 index 0000000000..0270b45673 Binary files /dev/null and b/research/p104-findings-nov15/code/slack/sol_ctl_2.npy differ diff --git a/research/p104-findings-nov15/code/slack/sol_ctl_3.npy b/research/p104-findings-nov15/code/slack/sol_ctl_3.npy new file mode 100644 index 0000000000..0e93585718 Binary files /dev/null and b/research/p104-findings-nov15/code/slack/sol_ctl_3.npy differ diff --git a/research/p104-findings-nov15/code/slack/sol_ctl_4.npy b/research/p104-findings-nov15/code/slack/sol_ctl_4.npy new file mode 100644 index 0000000000..637fe6fc65 Binary files /dev/null and b/research/p104-findings-nov15/code/slack/sol_ctl_4.npy differ diff --git a/research/p104-findings-nov15/code/slack/sol_ctl_5.npy b/research/p104-findings-nov15/code/slack/sol_ctl_5.npy new file mode 100644 index 0000000000..4c6af04fde Binary files /dev/null and b/research/p104-findings-nov15/code/slack/sol_ctl_5.npy differ diff --git a/research/p104-findings-nov15/code/slack/step1.py b/research/p104-findings-nov15/code/slack/step1.py new file mode 100644 index 0000000000..7925b86d3a --- /dev/null +++ b/research/p104-findings-nov15/code/slack/step1.py @@ -0,0 +1,40 @@ +"""DYINGPAIR step 1 [C-param]: price two-arm B layouts (L=17, nl=8) where c own roles entering left stage k die after stage d (< nl-1). +Caps: 'lemma' |S_k|-4 with S_k = roles alive on both sides of the boundary (NONNEST capS). Penalty variant: DYINGGEN Lemma 2 adds |E|-1 +extra roles (here 1 for a pair, entering at d+1 and living to the centre); Floor' check |Z_j| >= 2(j+1)+2-b_j with b_j <= #dead roles.""" +import sys; sys.path.insert(0,'.') +from layout_price import invoice, price, two_arm +L=17; nl=8 +Zof=lambda ms:[sum(1 for m in ms if k in m) for k in range(L)] +def capS(ms): + cap=[] + for k in range(L-1): + if k==nl-1: cap.append(0); continue + S=sum(1 for m in ms if k in m and k+1 in m); cap.append(max(S-4,0)) + return cap +def dying(specs): + """specs: list of (arm, k, d, c, extra): in arm 'L' (stages 0..nl-1) or 'R' (mirror), c of the roles entering at arm-depth k now live k..d; extra roles enter at d+1.""" + ms=two_arm() + for arm,k,d,c,extra in specs: + if arm=='L': full=set(range(k,nl)); new=set(range(k,d+1)); ex=set(range(d+1,nl)) + else: full=set(range(nl,L-k)); new=set(range(L-1-d,L-k)); ex=set(range(nl,L-1-d)) + for _ in range(c): ms.remove(full); ms.append(set(new)) + for _ in range(extra): ms.append(set(ex)) + return ms +def floor_ok(ms, ndead): + Z=Zof(ms); return all(Z[j]>=2*(j+1)+2-ndead for j in range(nl)) +def go(lab, specs, ndead): + ms=dying(specs); T,P,Z=invoice(ms,L,capS(ms)); r=price(T,P,lab+(' FLOOR-FAIL' if not floor_ok(ms,ndead) else ''),quiet=False); return r +B=go('B control', [], 0) +best=[] +for c in (1,2): + for k in range(0,nl-2): + for d in range(k,nl-2): + for extra in ((0,1) if c==2 else (0,)): + r=go('%s k=%d d=%d extra=%d (one arm)'%('pair' if c==2 else 'single',k,d,extra),[('L',k,d,c,extra)],c) + best.append((min(x/y for x,y in zip(r,B)),c,k,d,extra)) +print('--- per-layout min ratio over 3 columns vs B, >= 1.005 passes:') +for t in sorted(best,reverse=True)[:12]: print(' %.4f c=%d k=%d d=%d extra=%d'%t) +print('pass count', sum(1 for t in best if t[0]>=1.005), 'of', len(best)) +for (k,d) in ((0,3),(1,4),(0,1),(1,2)): + for extra in (0,1): + r=go('pair per arm k=%d d=%d extra=%d'%(k,d,extra),[('L',k,d,2,extra),('R',k,d,2,extra)],2) diff --git a/research/p104-findings-nov15/code/slack/t1.py b/research/p104-findings-nov15/code/slack/t1.py new file mode 100644 index 0000000000..50af7d370c --- /dev/null +++ b/research/p104-findings-nov15/code/slack/t1.py @@ -0,0 +1,27 @@ +"""SLACK t1 [N rs3]: L=4, left arm stages 0,1,2 (centre 2), right arm stage 3. Lone role e on [0,1] (dies before the centre). +Rows: ctl = nested B(3+1); e = + lone role; eL01 = + 1 line over [0,1] on Z0 cap Z1 (incl e: cap |S|-4 = 1, allowed); +TEST r01 = + 1 line over [0,1] supported on Z_0 \\ e (needs rank(Y0,Y1|Z0\\e) <= 3 AND X-side); TEST r02 = 1 line over [0,2] on Z_0\\e.""" +import os; os.environ['OMP_NUM_THREADS']='1'; os.environ['OPENBLAS_NUM_THREADS']='1'; os.environ['VECLIB_MAXIMUM_THREADS']='1' +import sys, time; sys.path.insert(0,'.') +import numpy as np +from rs3 import Model, run +L=4; nl=3 +def masks(e=True): + ms=[set(range(L))]*2 + [set(range(k,nl)) for k in range(nl) for _ in range(2)] + [{3},{3}] + if e: ms=ms+[{0,1}] + return ms +def build(ms): + n=len(ms); Z=[[r for r,s in enumerate(ms) if k in s] for k in range(L)]; th=n+2-2*L + return n,[('S',Z[k]) for k in range(nl)]+[('T',)]*th+[('S',Z[k]) for k in range(nl,L)],Z +row=sys.argv[1]; starts=int(sys.argv[2]); nfev=int(sys.argv[3]) if len(sys.argv)>3 else 1500 +ms=masks(row!='ctl'); n,o,Z=build(ms); e=n-1 +B0=[r for r in Z[0] if r!=e] +lines={'ctl':(),'e':(),'eL01':(((0,1),1,None),),'r01':(((0,1),1,B0),),'r02':(((0,1,2),1,B0),),'r12':(((1,2),1,None),)}[row] +m=Model(n,o,-0.5,lines) +print(row,'n',n,'Z',[len(z) for z in Z],'dim',m.dim,flush=True) +t0=time.time(); hits=0 +for i in range(starts): + h,b,z=run(m,1,seed=1000+i,nfev=nfev); hits+=h + print(' start %d solved %d best %.1e (%.0fs)'%(i,h,b,time.time()-t0),flush=True) + if h: np.save('sol_%s_%d.npy'%(row,i),z) +print(row,'solved',hits,'/',starts) diff --git a/research/p104-findings-nov15/code/slack/t2.py b/research/p104-findings-nov15/code/slack/t2.py new file mode 100644 index 0000000000..dc70a9787a --- /dev/null +++ b/research/p104-findings-nov15/code/slack/t2.py @@ -0,0 +1,27 @@ +"""SLACK t1 [N rs3]: L=4, left arm stages 0,1,2 (centre 2), right arm stage 3. Lone role e on [0,1] (dies before the centre). +Rows: ctl = nested B(3+1); e = + lone role; eL01 = + 1 line over [0,1] on Z0 cap Z1 (incl e: cap |S|-4 = 1, allowed); +TEST r01 = + 1 line over [0,1] supported on Z_0 \\ e (needs rank(Y0,Y1|Z0\\e) <= 3 AND X-side); TEST r02 = 1 line over [0,2] on Z_0\\e.""" +import os; os.environ['OMP_NUM_THREADS']='1'; os.environ['OPENBLAS_NUM_THREADS']='1'; os.environ['VECLIB_MAXIMUM_THREADS']='1' +import sys, time; sys.path.insert(0,'.') +import numpy as np +from rs3 import Model, run +L=4; nl=int(os.environ.get("NL","3")) +def masks(e=True): + ms=[set(range(L))]*2 + [set(range(k,nl)) for k in range(nl) for _ in range(2)] + [set(range(nl,k+1)) for k in range(L-1,nl-1,-1) for _ in range(2)] + if e: ms=ms+[{0,1}] + return ms +def build(ms): + n=len(ms); Z=[[r for r,s in enumerate(ms) if k in s] for k in range(L)]; th=n+2-2*L + return n,[('S',Z[k]) for k in range(nl)]+[('T',)]*th+[('S',Z[k]) for k in range(nl,L)],Z +row=sys.argv[1]; starts=int(sys.argv[2]); nfev=int(sys.argv[3]) if len(sys.argv)>3 else 1500 +ms=masks(row!='ctl'); n,o,Z=build(ms); e=n-1 +B0=[r for r in Z[0] if r!=e] +lines={'ctl':(),'e':(),'eL01':(((0,1),1,None),),'r01':(((0,1),1,B0),),'r02':(((0,1,2),1,B0),),'r12':(((1,2),1,None),),'r01x':(((0,1),2,B0),)}[row] +m=Model(n,o,-0.5,lines) +print(row,'n',n,'Z',[len(z) for z in Z],'dim',m.dim,flush=True) +t0=time.time(); hits=0 +for i in range(starts): + h,b,z=run(m,1,seed=1000+i,nfev=nfev); hits+=h + print(' start %d solved %d best %.1e (%.0fs)'%(i,h,b,time.time()-t0),flush=True) + if h: np.save('sol_%s_%d.npy'%(row,i),z) +print(row,'solved',hits,'/',starts) diff --git a/research/p104-findings-nov15/code/slack/win.py b/research/p104-findings-nov15/code/slack/win.py new file mode 100644 index 0000000000..bd38fdd827 --- /dev/null +++ b/research/p104-findings-nov15/code/slack/win.py @@ -0,0 +1,40 @@ +"""SLACK win.py [exact, sympy]: abstract window block. Units u0,u1 (stage A, earlier), u2,u3 (stage B). Gram G = y_i . x_j from Lemma S: +G[i,j] = (W Z^T)_ij if i later than j, -(W Z^T)_ij if earlier, stage block G_ss = [[0,g],[h,0]] with W_s Z_s^T = [[1,-g],[h,1]]. +Prefix action on coefficients beta (v = sum beta_j x_j): T_i: beta -> beta - 2 e_i (G[i,:] beta), applied u0,u1,u2,u3. +A b_K>0 block needs A beta = -beta with Z^T beta = 0 (v orthogonal to all later y, NESTPROOF L1/L3).""" +import sympy as sp +g0,h0,g1,h1=sp.symbols('g0 h0 g1 h1') +z=sp.symbols('z0:8') +Z0=sp.Matrix(2,2,z[:4]); Z1=sp.Matrix(2,2,z[4:]) +K0=sp.Matrix([[1,-g0],[h0,1]]); K1=sp.Matrix([[1,-g1],[h1,1]]) +W0=K0*Z0.T.inv(); W1=K1*Z1.T.inv() +W=sp.Matrix.vstack(W0,W1); Zm=sp.Matrix.vstack(Z0,Z1); WZ=W*Zm.T +G=sp.zeros(4,4) +for i in range(4): + for j in range(4): + si,sj=i//2,j//2 + if si>sj: G[i,j]=WZ[i,j] + elif si= W['public']) +print('ports*(phase - sign) =', tot, ' owned/call', own, ' S =', tot - L*own, ' (PRICEKIT S', base['S'], ')') +# D0 split +def d0(h): return sum(v*r*math.log(r) for r, v in h.items() if r > 1) +mm = Counter(); [mm.update(v) for v in nat['main_moves'].values()] +print('D0 parts per invoice: caller T %.1f caller P %.1f native helpers T %.1f mains P %.1f other native P %.1f' % ( + d0(ci['T']), d0(ci['P']), L*d0(nat['T']), L*d0(mm), L*(d0(nat['P']) - d0(mm)))) +def prize(label, dT=None, dP=None, cT=None, cP=None): + T = Counter(ci['T']); P = Counter(ci['P']) + for r, v in (cT or {}).items(): T[r] += v + for r, v in (cP or {}).items(): P[r] += v + c2 = dict(ci); c2['T'] = +T; c2['P'] = +P + n2 = pk.native_param(nat, 96, 8, dT=dT, dP=dP) + pr = pk.price(pk.invoice(c2, n2), 30); k = pr['kappa']['lo'] + print('%-58s S %5d kappa %s x%.5f' % (label, pr['S'], pk.kstr(pr['kappa'], 14), float(k / k0))) +prize('(a) rho 2->1 on A: one QUERY less per A port', cP={1: -96}) +prize('(a2) rho 2->1 on A and B (B: one IDLE_ODD -> IDLE_ONE)', cP={1: -96, 9: -8, 8: 8}) +prize('(b) owned c -1 (2 rank-1 owned jobs less per call)', dP={1: -2}) +prize('(c) D-bound: every main paid as ONE rank-8 MOVE', dP={**{r: -v for r, v in mm.items()}, 8: 400}) +prize('(d) one extra odd field per A port, T[9]+P[9] (neutral)', cT={9: 96}, cP={9: 96}) +prize('(e) one extra phase unit per A port per call (+17/port)', cT={1: 17*96}) diff --git a/research/p104-findings-nov15/code/spec/chart.py b/research/p104-findings-nov15/code/spec/chart.py new file mode 100644 index 0000000000..28d0c0f9cf --- /dev/null +++ b/research/p104-findings-nov15/code/spec/chart.py @@ -0,0 +1,29 @@ +# Exact: X starts on , Y ends on q_p^perp, q_p.q_p = 1 => no chart F with <= F <= q_p^perp (X and Y never share a chart +# that is both above X-birth and below Y-end). Also: which charts X and Y actually share during the word. +import sys +sys.path.insert(0, '../../PRICEKIT') +import pricekit as pk +W = pk.load('../../ATTACK/STACK7616/best7616.json.gz') +def span(b): + s = {0} + for v in b: s |= {x ^ v for x in s} + return frozenset(s) +sp = [span(b) for b in W['frames']] +dot = lambda a, b: bin(a & b).count('1') & 1 +ok_line = ok_perp = aniso = 0 +for i, r in enumerate(W['live']): + q = W['ports'][i] + ok_line += sp[W['initial'][r]] == frozenset({0, q}) + ok_perp += sp[W['final'][r + 112]] == frozenset(v for v in range(512) if not dot(v, q)) + aniso += dot(q, q) +print('ports', len(W['live']), 'X starts on :', ok_line, ' Y ends on q_p^perp:', ok_perp, ' q_p.q_p = 1:', aniso) +X2Y2 = sum(sp[W['initial'][x]] == frozenset({0, W['ports'][W['live'].index(l)]}) for x, l in + zip(W['hosted']['X2'], [W['live'][i] for i in W['hosted']['Dports']])) +print('X2 starts on its port line (assuming X2 order = Dports order):', X2Y2, 'of', len(W['hosted']['X2'])) +# direct X<->Y same-chart ADDs on the same port +cur = list(W['initial']) + [None] * W['owned_fields']; pair = 0; cross = 0 +partner = {r: r + 112 for r in W['live']}; partner.update({r + 112: r for r in W['live']}) +for e in W['program']: + if e[0] == 'MOVE': cur[e[1]] = e[3] + elif e[0] in ('ADD', 'COPY') and partner.get(e[1]) == e[2]: pair += 1 +print('direct ADDs between a port\'s own X and Y:', pair) diff --git a/research/p104-findings-nov15/code/spec/inv.py b/research/p104-findings-nov15/code/spec/inv.py new file mode 100644 index 0000000000..e00ffd113b --- /dev/null +++ b/research/p104-findings-nov15/code/spec/inv.py @@ -0,0 +1,20 @@ +import sys +sys.path.insert(0, '../PRICEKIT') +import pricekit as pk +from collections import Counter +W = pk.load('../ATTACK/STACK7616/best7616.json.gz') +C = pk.load('../MONOCALLER/7113_CALLER.json') +nat = pk.native_from_word(W); ci = pk.caller_invoice(C) +inv = pk.invoice(ci, nat); pr = pk.price(inv, 30) +print('kappa', pk.kstr(pr['kappa']), 'S', pr['S'], 'D0', float(pr['D0'])) +mass = lambda h: sum(r*v for r, v in h.items()) +T, P = nat['T'], nat['P'] +print('native per call: T mass', mass(T), 'P mass', mass(P), 'helpers', sum(T.values()), 'S_nat', mass(T)-mass(P)) +print('main moves mass', {k: mass(v) for k, v in nat['main_moves'].items()}) +n = W['public']; dim = [len(b) for b in W['frames']] +om = Counter() +for e in W['program']: + if e[0] == 'MOVE' and e[1] >= n: om[abs(dim[e[3]] - dim[e[2]])] += 1 +print('owned move hist', dict(om), 'mass', mass(om)) +print('calls', ci['nJ'], 'caller T mass', mass(ci['T']), 'caller P mass', mass(ci['P'])) +print('caller P', dict(ci['P'])); print('caller T', dict(ci['T'])) diff --git a/research/p104-findings-nov15/code/spec/owned.py b/research/p104-findings-nov15/code/spec/owned.py new file mode 100644 index 0000000000..f09dbe9fcc --- /dev/null +++ b/research/p104-findings-nov15/code/spec/owned.py @@ -0,0 +1,34 @@ +# Which public classes does each owned lifetime feed, and what does it pay? (#7617) +import sys, resource +sys.path.insert(0, '../../PRICEKIT') +import pricekit as pk +from collections import Counter, defaultdict +W = pk.load('../../ATTACK/STACK7616/best7616.json.gz') +n = W['public']; H = W['hosted']; dim = [len(b) for b in W['frames']] +cls = {} +for r in W['live']: cls[r] = 'X'; cls[r + 112] = 'Y' +for r in H['X2']: cls[r] = 'X2' +for r in H['Y2']: cls[r] = 'Y2' +act = {W['live'][i] for i in H['Dports']} +life = {}; lives = [] +for i, e in enumerate(W['program']): + op, r = e[0], e[1] + if op == 'INIT': + life[r] = dict(start=i, frame=dim[e[2]], pay=0, up=0, feeds=Counter(), reads=Counter(), ownfeed=0); lives.append(life[r]) + elif op == 'CLEAR': + life[r]['end'] = dim[e[2]]; del life[r] + elif op == 'MOVE' and r >= n: + d = dim[e[3]] - dim[e[2]]; life[r]['pay'] += abs(d); life[r]['up'] += max(d, 0) + elif op in ('ADD', 'COPY'): + s = e[2] + if s >= n and r < n: life[s]['feeds'][cls.get(r, 'H')] += 1 + if s >= n and r >= n: life[s]['ownfeed'] += 1 + if r >= n and s < n: life[r]['reads'][cls.get(s, 'H')] += 1 +print('owned lifetimes', len(lives), 'paying', sum(1 for L in lives if L['pay']), 'mass', sum(L['pay'] for L in lives)) +agg = Counter(); aggn = Counter() +for L in lives: + if not L['pay']: continue + key = ('feeds:' + '+'.join(sorted(L['feeds'])) or '-') + ' | reads:' + '+'.join(sorted(L['reads'])) + agg[key] += L['pay']; aggn[key] += 1 +for k, v in agg.most_common(): print('%4d mass %3d lives %s' % (v, aggn[k], k)) +print('rss MB', resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 2**20) diff --git a/research/p104-findings-nov15/code/spec/park7617.py b/research/p104-findings-nov15/code/spec/park7617.py new file mode 100644 index 0000000000..3827eefd57 --- /dev/null +++ b/research/p104-findings-nov15/code/spec/park7617.py @@ -0,0 +1,153 @@ +# R2 PoC: stay drop on #7617 (BIG/TAILS engine) + exact owned-park cleanup of the leftover dirty main rows at their final +# chart G, using rows ever resident at G (forward park: INIT at G, accumulate, deliver after the main's last read, dirty CLEAR). +import os,sys,json,gzip,threading,resource,time,collections as C +from fractions import Fraction as Q +os.environ['SUP_BASE']='inputs/ATTACK/STACK7616/best7616.json.gz' +def _w(): + while True: + if resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1e9>3.8: print('RSS bail',flush=True); os._exit(3) + time.sleep(1) +threading.Thread(target=_w,daemon=True).start() +HERE=os.path.dirname(os.path.abspath(__file__));STAYS=json.loads(sys.argv[1]);outtag=sys.argv[2] +os.chdir('inputs/BIG/TAILS/code');sys.path.insert(0,'.') +from k3 import * +import circ +W8=json.load(gzip.open('inputs/ATTACK/PAIR/code/w7608.json.gz','rt')); nb8=circ.Base(W8) +def mapst(st,old): + h,F,a,d=st; L=old.stays[h]; i=[x[1] for x in L].index(a) + pf=L[i-1][0]; nf=L[i+1][0]; N=nb.stays[h] + c=[j for j,x in enumerate(N) if 0coef) + def reduce(v,cmb): + v=dict(v);cmb=dict(cmb) + for pc,b,bc in basis: + c=v.get(pc) + if c: + for k,x in b.items(): + z=v.get(k,0)-c*x + if z:v[k]=z + else:v.pop(k,None) + for k,x in bc.items(): + z=cmb.get(k,0)-c*x + if z:cmb[k]=z + else:cmb.pop(k,None) + return v,cmb + for k,(i,r,row) in enumerate(snaps): + v,cmb=reduce(row,{k:Q(1)}) + if v: + pc=min(v);c=v[pc];v={a:b/c for a,b in v.items()};cmb={a:b/c for a,b in cmb.items()} + basis.append((pc,v,cmb)) + v,cmb=reduce(J,{}) + if v:return None,len(basis) + return {k:-x for k,x in cmb.items()},len(basis) # sum mu_k row_k = J +def dyadic(q):return q.denominator&(q.denominator-1)==0 and abs(q)+1<2**16 +for st in STAYS: + t0=time.time();m=mapst(tuple(st[0]),nb8);mode=st[1] + r=test([m],frames_mode=mode,K=300 if mode=='all' else 4000,owned=False,want=True) + ed,W2,org,seeds,sv=make([m]) + evs=[] + bad=False + for (a,b,pos,F),v in zip(r.get('cands',[]),r.get('x',[])): + if v: + q=ratrec(v) + if q is None:bad=True;break + evs.append((pos,['ADD',a,b,str(q),F])) + if bad:print(st,'ratrec fail');continue + W3=insert(W2,evs);n3=W3['public'];E=expected(B0) + rows,_,_=xreplay(W3,set()) + dirty=[p for p in range(n3) if rows[p]!=E[p]] + G={p:W3['final'][p] for p in dirty} + _,_,lr0=xreplay(W3,set()) + A={} + for p in dirty: + c=W3['initial'][p];hop=None + for i,x in enumerate(W3['program']): + if x[0]=='MOVE' and x[1]==p: + if ilastread.get(p,-1)] + a,hop=A[p] + S=[s for s in snaps[G[p]] if s[1]!=p] + if hop is not None:S=S+[s for s in snaps[a] if s[1]!=p and s[0]3:S=S+[s for s in snaps[a] if s[1]!=p] # diagnostic: time-ignored + print(' p',p,'a',a,'hop',hop,'lastread',lastread.get(p)) + mu,rk=solve(S,J) + if mu is None:print(' p',p,'NOT in span; rank',rk,'support',len(J));ok=False;break + nd=[x for x in mu.values() if not dyadic(x)] + print(' p',p,'G',G[p],'|J|',len(J),'rank',rk,'uses',len(mu),'snapshots; non-dyadic',len(nd),'moves after last read',len(moves_after)) + plan.append((p,G[p],[(S[k][0],S[k][1],x) for k,x in mu.items()])) + if not ok:continue + # build: owned park per dirty main; label free over [first use, end] + prog=W3['program'];L=len(prog);n=W3['public'];NO=W3['owned_fields'] + busy=C.defaultdict(list);open_={} + for i,e in enumerate(prog): + if e[0]=='INIT':open_[e[1]]=i + elif e[0]=='CLEAR':busy[e[1]].append((open_.pop(e[1]),i)) + ins=C.defaultdict(list);tail=[];used=[] + for p,Gp,uses in plan: + t0_=min(i for i,_,_ in uses) + lab=next((o for o in range(n,n+NO) if o not in used and all(b=0:newp.append(prog[i]) + newp.extend(ins.get(i,[])) + W5=dict(W3);W5['program']=newp+tail + rows5,_,_=xreplay(W5,set()) + bad5=[p for p in range(n) if rows5[p]!=E[p]] + print(' parks',len(plan),'labels',used,'added events',len(newp)+len(tail)-L,'rows differing after parks',bad5,round(time.time()-t0,1),flush=True) + if not bad5: + fn=os.path.join(HERE,'hit_%s_%s.json.gz'%(outtag,'_'.join(map(str,m)))) + with gzip.GzipFile(fn,'wb',mtime=0) as f:f.write(json.dumps(W5,separators=(',',':')).encode()) + print(' SAVED',fn,flush=True) diff --git a/research/p104-findings-nov15/code/spec/paths.py b/research/p104-findings-nov15/code/spec/paths.py new file mode 100644 index 0000000000..66be7477c7 --- /dev/null +++ b/research/p104-findings-nov15/code/spec/paths.py @@ -0,0 +1,63 @@ +import gzip, json, sys +from collections import Counter, defaultdict +W = json.load(gzip.open(sys.argv[1], 'rt')) +n = W['public']; N = n + W['owned_fields'] +fr = [frozenset() for _ in W['frames']] +def span(b): + s = {0} + for v in b: s |= {x ^ v for x in s} + return frozenset(s) +sp = [span(b) for b in W['frames']]; dim = [len(b) for b in W['frames']] +def meetdim(a, b): return len(sp[a] & sp[b]).bit_length() - 1 +def d(a, b): return dim[a] + dim[b] - 2 * meetdim(a, b) +# build per-lifetime stop lists: [chart, nops] +cur = list(W['initial']) + [None] * W['owned_fields'] +life = defaultdict(list) # key (reg, lifeno) -> list of [chart, ops] +lno = Counter() +for r in range(n): life[(r, 0)].append([cur[r], 0]) +for e in W['program']: + op, r = e[0], e[1] + if op == 'MOVE': + life[(r, lno[r])].append([e[3], 0]) + elif op == 'INIT': + lno[r] += 1; life[(r, lno[r])].append([e[2], 0]) + elif op == 'CLEAR': + life[(r, lno[r])][-1][1] += 1 # count the clear as an op at that chart + elif op in ('ADD', 'COPY'): + life[(r, lno[r])][-1][1] += 1 + s = e[2]; life[(s, lno[s])][-1][1] += 1 + elif op == 'SCALE': + life[(r, lno[r])][-1][1] += 1 +# analysis +tot_turn = Counter(); empty_mono = 0; rows = [] +for (r, l), stops in life.items(): + owned = r >= n + for i in range(1, len(stops) - 1): + a, b, c = stops[i-1][0], stops[i][0], stops[i+1][0] + if stops[i][1] == 0: + r1, r2, rf = d(a, b), d(b, c), d(a, c) + mono = (sp[a] < sp[b] < sp[c]) or (sp[a] > sp[b] > sp[c]) + if mono: empty_mono += 1 + rows.append((owned, r, i, dim[a], dim[b], dim[c], r1, r2, rf, mono)) +print('lifetimes', len(life), 'empty intermediate stops', len(rows), 'monotone', empty_mono) +for x in rows[:60]: print(x) +# helper hop statistics and per-helper D0 cost f(credit) - sum f(hops) +import math +f = lambda r: r*math.log(r) if r > 0 else 0.0 +main = set(W['live']) | {r+112 for r in W['live']} | set(W['hosted']['X2']) | set(W['hosted']['Y2']) +hopsc = Counter(); noops = 0; cost = 0.0; firstops = Counter() +for r in range(n): + if r in main: continue + st = life[(r, 0)] + hops = [d(st[i-1][0], st[i][0]) for i in range(1, len(st))] + hopsc[len(hops)] += 1 + if all(s[1] == 0 for s in st): noops += 1 + cost += f(sum(hops)) - sum(f(h) for h in hops) + firstops[(st[0][1] > 0, st[-1][1] > 0)] += 1 +print('helper hop-count hist', sorted(hopsc.items()), 'helpers with no ops', noops) +print('helper D0 cost per call', round(cost, 2), '(ops at first stop, ops at last stop) hist', dict(firstops)) +for r in range(n): + if r in main: continue + st = life[(r, 0)] + if st[0][1] == 0 or st[-1][1] == 0: + print('helper', r, [(dim[c], k) for c, k in st]) diff --git a/research/p104-findings-nov15/code/spec/relaysw.py b/research/p104-findings-nov15/code/spec/relaysw.py new file mode 100644 index 0000000000..47d9188980 --- /dev/null +++ b/research/p104-findings-nov15/code/spec/relaysw.py @@ -0,0 +1,98 @@ +# R2 class (c): RELAY frames on #7617. For every pinned transfer j->i at a relay frame F (window as BIG/RELAY pin.py), +# test the SANDWICH condition for any fresh-register (clean-ancilla-must-uncompute) network, j the only holder of its row: +# exists t1=NP or r in mains +dim=[len(b) for b in W['frames']] +def span(b): + s={0} + for v in b:s|={x^v for x in s} + return frozenset(s) +SP=[span(b) for b in W['frames']] +le=lambda a,b:SP[a]<=SP[b] +f=lambda r:r*math.log(r) if r>0 else 0.0 +def evframe(e): + if e[0] in('ADD','COPY'):return e[4] + if e[0]=='SCALE':return e[3] + if e[0] in('INIT','CLEAR'):return e[2] +ch={r:[(W['initial'][r],-1)] for r in range(NP)} +for i,e in enumerate(P): + if e[0]=='MOVE' and e[1]max(a,lo)] + Sj=[(max(a,lo),min(b,hi),z) for a,b,z in vstays(j,F) if z!=F and min(b,hi)>max(a,lo)] + for a,b,u in Si: + if park: + if any(z==u and zat and isblk(u)] + lo=max(before) if before else -1;hi=min(after) if after else len(P) + co=False + for a,b,G in ST[i]: + if G==F or b<=lo or a>=hi:continue + if any(z==G and zamax(a,lo) for za,zb,z in ST[j]):co=True;break + if co:stat['not pinned']+=1;continue + npin+=1;s=sandwich(i,j,lo,hi,F);p=sandwich(i,j,lo,hi,F,park=True) + stat[('pinned','sandwich' if s else 'no-sandwich','park' if p else 'no-park')]+=1 + allok&=s;allpark&=p + frames.append((F,gain[F],npin,allok,allpark,endF[F])) +print(dict(stat)) +ok=[x for x in frames if x[3]];okp=[x for x in frames if x[4]] +print('frames all-pinned sandwich-feasible',len(ok),'gain %.1f'%sum(x[1] for x in ok),'(of which F is no helper endpoint:',sum(1 for x in ok if x[5]==0),'gain %.1f)'%sum(x[1] for x in ok if x[5]==0)) +print('frames all-pinned park-feasible',len(okp),'gain %.1f'%sum(x[1] for x in okp)) +print('frames with 0 pinned transfers',sum(1 for x in frames if x[2]==0)) +json.dump(frames,open('relaysw.json','w')) diff --git a/research/p104-findings-nov15/code/spec/shuttle.py b/research/p104-findings-nov15/code/spec/shuttle.py new file mode 100644 index 0000000000..9d08a86b06 --- /dev/null +++ b/research/p104-findings-nov15/code/spec/shuttle.py @@ -0,0 +1,51 @@ +"""[C-param] SPECBREAK shuttle.py: a new 1-hop public register is kappa-free; a k-hop one costs the hop-split D0 (new-public-register pricing). +Own code + pricekit.py (vendored from #7607). Run here on the #7546 word (data/w7546.json.gz) under the #7113 caller; the lane ran it on the +frontier word. Asserts: replay ok, kappa of the 1-hop shuttle equal to the base kappa to 34 digits, S unchanged, and dD0 of the +0->4->9 helper equal to 17 (f(9) - f(4) - f(5)) exactly (f = r ln r). Prints PASS_SHUTTLE.""" +import sys, copy, math, os +_HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.join(_HERE, '..')) +import pricekit as pk +W0 = pk.load(os.path.join(_HERE, '..', '..', 'data', 'w7546.json.gz')); C = pk.load(os.path.join(_HERE, '..', '..', 'data', '7113_CALLER.json.gz')) +ci = pk.caller_invoice(C) +def kap(W): + inv = pk.invoice(ci, pk.native_from_word(W)); pr = pk.price(inv, 40); return pr +def add_public(W, chain): + """append public register h=n with charts chain[0] -> ... -> chain[-1], MOVEs at program end; owned shifted +1""" + W = copy.deepcopy(W); n = W['public'] + sh = lambda r: r + 1 if r >= n else r + prog = [] + for e in W['program']: + e = list(e); e[1] = sh(e[1]) + if e[0] in ('ADD', 'COPY'): e[2] = sh(e[2]) + prog.append(e) + for a, b in zip(chain, chain[1:]): prog.append(['MOVE', n, a, b, False]) + W['program'] = prog; W['public'] = n + 1 + W['initial'] = W['initial'] + [chain[0]]; W['final'] = W['final'] + [chain[-1]] + return W +dim = [len(b) for b in W0['frames']] +sp = [frozenset() for _ in dim] +def span(b): + s = {0} + for v in b: s |= {x ^ v for x in s} + return frozenset(s) +sp = [span(b) for b in W0['frames']] +# pick a nested chain 0-dim -> 4-dim -> 9-dim from existing frames +c0 = dim.index(0); c9 = dim.index(9) +c4 = next(i for i in range(len(dim)) if dim[i] == 4) +base = kap(W0) +print('base kappa', pk.kstr(base['kappa'], 34), 'S', base['S'], 'D0', base['D0']) +res = [] +for chain in ([c4, c9], [c0, c4, c9]): + W = add_public(W0, chain); rp = pk.replay(W) + pr = kap(W); res.append((rp, pr)) + print('chain dims', [dim[c] for c in chain], 'replay ok', rp['ok'], 'public', W['public'], + 'kappa', pk.kstr(pr['kappa'], 34), 'S', pr['S'], 'dD0 total', pr['D0'] - base['D0']) +f = lambda r: r * math.log(r) +print('predicted dD0 for (4,5) split, x17 calls:', 17 * (f(9) - f(4) - f(5))) + +(rp1, pr1), (rp2, pr2) = res +assert rp1['ok'] and rp2['ok'] +assert pk.kstr(pr1['kappa'], 34) == pk.kstr(base['kappa'], 34) and pr1['S'] == base['S'] and abs(float(pr1['D0'] - base['D0'])) < 1e-9 +assert pr2['S'] == base['S'] and abs(float(pr2['D0'] - base['D0']) - 17 * (f(9) - f(4) - f(5))) < 1e-6 +print('PASS_SHUTTLE') diff --git a/research/p104-findings-nov15/code/spec/splitbound.py b/research/p104-findings-nov15/code/spec/splitbound.py new file mode 100644 index 0000000000..5c557e8c17 --- /dev/null +++ b/research/p104-findings-nov15/code/spec/splitbound.py @@ -0,0 +1,68 @@ +# R2 class (a): split jobs on #7617 (BIG/LEAKPARK c7617.pkl). Sandwich lemma => a fresh-register network can supply the +# G-pure row e_g at (F,t) only from g's row-clean windows (before its first write, after its last write) at charts z1<=F<=z2, +# at network cost >= f(z2-z1)-f(F-z1)-f(z2-F) (one fresh helper z1->F->z2; 0 if z1==F or z2==F). Upper bound on the +# class: per helper, best over its jobs of tau(a,b) - that cost (ignores the column network, c.d pairing, pool rules). +import gzip,json,pickle,math,collections as C +W=json.load(gzip.open('../../ATTACK/STACK7616/best7616.json.gz','rt'));n=W['public'];P=W['program'] +dim=[len(b) for b in W['frames']] +def span(b): + s={0} + for v in b:s|={x^v for x in s} + return frozenset(s) +SP=[span(b) for b in W['frames']] +f=lambda r:r*math.log(r) if r>0 else 0.0 +tau=lambda a,b:f(a+b)-f(a)-f(b) +path={r:[(W['initial'][r],-1)] for r in range(n)};fw={};lw={};fr={};lrd={} +for i,e in enumerate(P): + if e[0]=='MOVE' and e[1]=0 and i>=lw.get(g,-1)]+[chart_at(g,lw.get(g,-1))] + # row-clean charts (early: before first write; late: from last write on) + z1s=[c for c in early if SP[c]<=SP[F]];z2s=[c for c in late if SP[F]<=SP[c]] + if not z1s or not z2s:cnt['no clean bracket']+=1;continue + z1=max(z1s,key=lambda c:dim[c]);z2=min(z2s,key=lambda c:dim[c]) + cost=0.0 if (z1==F or z2==F) else f(dim[z2]-dim[z1])-f(dim[F]-dim[z1])-f(dim[z2]-dim[F]) + net=g0-cost;cnt['bracket']+=1 + if cost==0:cnt['free bracket (g row-clean at F)']+=1 + if net>1e-9:cnt['net>0']+=1 + if net>best.get(g,(0,))[0]:best[g]=(net,F,dim[z1],dim[F],dim[z2],j['a'],j['b']) +print(dict(cnt));print('helpers with net>0',len(best),'sum of best nets (D0/call, upper bound) %.2f'%sum(v[0] for v in best.values())) +print(sorted(best.values(),reverse=True)[:8]) +# column-clean windows (dual): before first read, after last read +cc=C.Counter() +for j in J: + F=j['F'] + for g in j['helpers']: + early=[c for c,i in path[g] if i=lrd.get(g,-1)]+[chart_at(g,lrd.get(g,-1))] + cc[(any(c==F for c in early),any(c==F for c in late))]+=1 +print('column-clean at F (early,late):',dict(cc)) +# joint bound: need BOTH the row bracket (row-clean windows) and the column bracket (column-clean windows: before first +# read, after last read); network cost >= max(cost_row, cost_col) (generous: one network serving both). +def br(g,F,a_t,b_t): + early=[c for c,i in path[g] if i=0 and i>=b_t]+[chart_at(g,b_t)] + z1s=[c for c in early if SP[c]<=SP[F]];z2s=[c for c in late if SP[F]<=SP[c]] + if not z1s or not z2s:return None + z1=max(z1s,key=lambda c:dim[c]);z2=min(z2s,key=lambda c:dim[c]) + return 0.0 if (z1==F or z2==F) else f(dim[z2]-dim[z1])-f(dim[F]-dim[z1])-f(dim[z2]-dim[F]) +bj={};cj=C.Counter() +for j in J: + F=j['F'];g0=tau(j['a'],j['b']) + for g in j['helpers']: + cr=br(g,F,fw.get(g,10**9),lw.get(g,-1));cc_=br(g,F,fr.get(g,10**9),lrd.get(g,-1)) + if cr is None or cc_ is None:cj['missing bracket']+=1;continue + net=g0-max(cr,cc_);cj['both']+=1 + if net>1e-9: + cj['net>0']+=1 + if net>bj.get(g,(0,))[0]:bj[g]=(net,F,j['a'],j['b'],round(cr,2),round(cc_,2)) +print('JOINT',dict(cj),'helpers',len(bj),'sum %.2f D0/call'%sum(v[0] for v in bj.values()));print(sorted(bj.values(),reverse=True)[:10]) diff --git a/research/p104-findings-nov15/code/spec/stay7617.py b/research/p104-findings-nov15/code/spec/stay7617.py new file mode 100644 index 0000000000..e8587842bb --- /dev/null +++ b/research/p104-findings-nov15/code/spec/stay7617.py @@ -0,0 +1,37 @@ +# R2: run the BIG/TAILS stay-drop engine on #7617 (SUP_BASE=best7616) for given w7608 stays; save corrected word + dirty rows. +import os,sys,json,gzip,threading,resource,time,pickle +os.environ['SUP_BASE']='inputs/ATTACK/STACK7616/best7616.json.gz' +def _w(): + while True: + if resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1e9>3.8: print('RSS bail',flush=True); os._exit(3) + time.sleep(1) +threading.Thread(target=_w,daemon=True).start() +OUT=os.path.abspath(sys.argv[2]);todo=json.load(open(os.path.abspath(sys.argv[1])));lim=int(sys.argv[3]) +os.chdir('inputs/BIG/TAILS/code');sys.path.insert(0,'.') +from k3 import * +import circ +W8=json.load(gzip.open('inputs/ATTACK/PAIR/code/w7608.json.gz','rt')); nb8=circ.Base(W8) +def mapst(st,old): + h,F,a,d=st; L=old.stays[h]; i=[x[1] for x in L].index(a) + pf=L[i-1][0]; nf=L[i+1][0]; N=nb.stays[h] + c=[j for j,x in enumerate(N) if 0= a, after lastread(p)) [superset of the sandwich supply of ANY +# fresh-public-register network delivering with col_p final: Down(u,td) & Up(u,td), u in {a,G}, td>=lastread] +# J_p in PARK / FRESH? Stay feasible iff all dirty rows pass (owned/clear dirt reported separately). +import gzip,json,sys,collections as C,numpy as np,time,resource +from fractions import Fraction as Q +from flint import nmod_mat +P=2**31-1 +import threading,os +def _wd(): + while True: + if resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1e9>3.8:print('RSS>3.8GB bail',flush=True);os._exit(3) + time.sleep(1) +threading.Thread(target=_wd,daemon=True).start() +B=json.load(gzip.open('../../ATTACK/STACK7616/best7616.json.gz','rt'));dim=[len(b) for b in B['frames']] +def span(b): + s={0} + for v in b:s|={x^v for x in s} + return frozenset(s) +SP=[span(b) for b in B['frames']] +cache={} +def qm(s): + v=cache.get(s) + if v is None:q=Q(s);v=cache[s]=q.numerator%P*pow(q.denominator%P,P-2,P)%P + return v +def rank(rows): + if not rows:return 0 + A=np.array(rows,dtype=np.int64)%P;A=A[:,A.any(0)] + return nmod_mat(A.shape[0],A.shape[1],A.ravel().tolist(),P).rank() +def test(W,dirty): + n=W['public'];NO=W['owned_fields'];N=n+NO;prog=W['program'] + lastread={};mv=C.defaultdict(list) + for i,e in enumerate(prog): + if e[0] in('ADD','COPY'):lastread[e[2]]=i + elif e[0]=='MOVE':mv[e[1]].append((i,e[3])) + spec={} + for d in dirty: + if d['kind']!='pub':continue + p=d['r'];lr=lastread.get(p,-1);a=W['initial'][p];hop=None + for i,f in mv[p]: + if itl and c in upc[(p,k)]:v=X[r].astype(np.uint32) if v is None else v;UPk[(p,k)].append(v) + at=C.defaultdict(list) + for p in spec: + for k,(u,tl,th) in enumerate(WIN[p]):at[tl].append((p,k)) + def present(i): + for p,k in at.get(i,()): + for r in range(N): + if cur[r] in upc[(p,k)] and X[r].any():UPk[(p,k)].append(X[r].astype(np.uint32)) + for r in range(n):arrive(-1,r) + present(-1) + for i,e in enumerate(prog): + op,r=e[0],e[1] + if op=='MOVE':cur[r]=e[3];arrive(i,r) + elif op=='INIT':X[r]=0;cur[r]=e[2] + elif op=='CLEAR':X[r]=0;cur[r]=None + elif op=='SCALE':X[r]=X[r]*qm(e[2])%P + elif op=='COPY':X[r]=qm(e[3])*X[e[2]]%P + elif op=='ADD':X[r]=(X[r]+qm(e[3])*X[e[2]])%P + present(i) + out={} + for d in dirty: + if d['kind']!='pub':continue + p=d['r'];J=np.zeros(n,dtype=np.int64);J[d['cols']]=d['coef'] + pk=list({v.tobytes():v for v in PK[p]}.values()) + inter=[] + for k in range(len(WIN[p])): + A=basis(DN[(p,k)]);Bm=basis(UPk[(p,k)]) + inter+=intersect(A,Bm) + r0=rank(pk);r1=rank(pk+[J]);r2=rank(pk+inter);r3=rank(pk+inter+[J]) + out[p]=dict(park=r1==r0,fresh=r3==r2,rk=(r0,r2,len(inter)),hop=spec[p][2] is not None) + return out +def basis(rows): + if not rows:return np.zeros((0,1),dtype=np.int64) + A=np.array(list({v.tobytes():v for v in rows}.values()),dtype=np.int64)%P + M=nmod_mat(A.shape[0],A.shape[1],A.ravel().tolist(),P).rref()[0] + rk=0 + R=[] + for i in range(M.nrows()): + row=[int(M[i,j]) for j in range(M.ncols())] + if not any(row):break + R.append(row) + return np.array(R,dtype=np.int64).reshape(len(R),A.shape[1]) +def intersect(A,Bm): + if not len(A) or not len(Bm):return [] + M=np.vstack([A,(-Bm)%P]) # rows; find (alpha,beta) with alpha A - beta B... left nullspace + T=M.T.copy() + X_,nul=nmod_mat(T.shape[0],T.shape[1],T.ravel().tolist(),P).nullspace() + out=[] + for j in range(nul): + al=np.array([int(X_[i,j]) for i in range(len(A))],dtype=np.int64) + v=np.zeros(A.shape[1],dtype=np.int64) + for i in np.nonzero(al)[0]:v=(v+int(al[i])*A[i])%P + if v.any():out.append(v.astype(np.uint32)) + return out +L=[json.loads(l) for l in open(sys.argv[1]) if l.count('"kind": "pub"')<=6];lim=int(sys.argv[3]) if len(sys.argv)>3 else 10**9 +done=set() +try: + for l in open(sys.argv[2]):done.add(json.loads(l)['tag']) +except FileNotFoundError:pass +for x in [x for x in L if sum(d["kind"]=="pub" for d in x["dirty"])<=6][:lim]: + if x['tag'] in done:continue + t0=time.time() + W=json.load(gzip.open('outall/w3m_%s.json.gz'%x['tag'],'rt')) + o=test(W,x['dirty']) + kinds=C.Counter(d['kind'] for d in x['dirty']) + rec=dict(tag=x['tag'],st=x['st'],sv=x['sv'],kinds=dict(kinds),npub=len(o),park=all(v['park'] for v in o.values()), + fresh=all(v['fresh'] for v in o.values()),npark=sum(v['park'] for v in o.values()),nfresh=sum(v['fresh'] for v in o.values()), + detail={str(k):v for k,v in list(o.items())[:6]},sec=round(time.time()-t0,1),gb=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1e9) + open(sys.argv[2],'a').write(json.dumps(rec)+'\n') + print(rec['tag'],rec['kinds'],'park',rec['npark'],'fresh',rec['nfresh'],'/',rec['npub'],rec['sec'],'s',round(rec['gb'],2),'GB',flush=True) diff --git a/research/p104-findings-nov15/code/tails/bound.py b/research/p104-findings-nov15/code/tails/bound.py new file mode 100644 index 0000000000..9279d0fe51 --- /dev/null +++ b/research/p104-findings-nov15/code/tails/bound.py @@ -0,0 +1,66 @@ +import sys,json,collections +sys.path.insert(0,'.'); sys.path.insert(0,'inputs/PRICEKIT') +from tails import * +import pricekit as pk +fin=W['final'] +C='inputs/MONOCALLER/7113_CALLER.json' +nat=pk.native_from_word(W) +def kap(P): + n2=dict(nat); n2['P']=+P; return pk.run(C,n2,digits=20)['price']['kappa']['lo'] +K0=kap(collections.Counter(nat['P'])); print('base kappa',K0) +cj=json.load(open('inputs/ATTACK/PAIR/code/cands.json')) +byst={} +for cl in cj: + for x in cj[cl]: byst[tuple(x['st'])]=x +byrc=collections.defaultdict(list) +for k in byst: byrc[(k[0],k[1])].append(k) +recs=[] +for fn in sys.argv[1:]: + for l in open(fn): + o=json.loads(l) + if 'dirty' not in o: continue + st=tuple(o['st']) + if st not in byst: + c=byrc.get((st[0],st[1])) + if not c: continue + st=c[0] + if any(tuple(r['st'])==st for r in recs): continue + o['key']=st; recs.append(o) +print('stays with census',len(recs)) +def need(o): + req=[]; bad=[] + for d in o['dirty']: + if d['kind']!='pub': bad.append('owned'); continue + G=d['G'] + if G==127: continue + Tm=arrive(d['r'],G) + if d['ncols']>len(d['cols']): bad.append('trunc') + for c in d['cols']: + if fin[c]!=127: + if fin[c]==G or ssub(G,fin[c]): req.append((c,G,'main-ok',0.0)) + else: bad.append('main-col') + else: + s,dc=opt(c,G,Tm); req.append((c,G,s,dc)) + return req,bad +cls=collections.Counter(); rows=[] +for o in recs: + req,bad=need(o); o['req']=req; o['bad']=bad + nm=sum(1 for d in o['dirty'] if d['kind']=='pub' and d['G']!=127) + st=collections.Counter(s for _,_,s,_ in req) + legal=all(s in('ok','ok-atG','main-ok') for _,_,s,_ in req) + # chain consistency per helper (model A): required G set of each helper must be a chain + gs=collections.defaultdict(set) + for c,G,s,_ in req: gs[c].add(G) + chain=all(all(sub(a,b) or sub(b,a) for a in v for b in v) for v in gs.values()) + o['legal']=legal and not bad and nm>0; o['chainA']=chain and not bad and nm>0 + cls[('nomain' if nm==0 else 'main', 'bad' if bad else '', 'B-legal' if o['legal'] else ('A-chain' if o['chainA'] else 'A-conflict'))]+=1 + rows.append((o['key'],round(byst[o['key']]['m'],2),nm,len(req),dict(st),bad[:2],o['legal'],o['chainA'])) +for k,v in sorted(cls.items(),key=str): print(v,k) +json.dump([dict(key=o['key'],m=byst[o['key']]['m'],req=o['req'],bad=o['bad'],legal=o['legal'],chainA=o['chainA']) for o in recs],open('../need.json','w')) +# helper -> stays needing it, distribution +hn=collections.Counter(); hg=collections.defaultdict(set) +for o in recs: + for c,G,s,_ in o['req']: hg[c].add(G) +print('distinct (helper) required',len(hg),'helpers needed at >=2 incomparable G:',sum(1 for v in hg.values() if not all(sub(a,b) or sub(b,a) for a in v for b in v))) +print('status totals',collections.Counter(s for o in recs for _,_,s,_ in o['req'])) +print('per-stay #required pairs: quantiles',sorted(len(o['req']) for o in recs)[::max(1,len(recs)//10)]) diff --git a/research/p104-findings-nov15/code/tails/build.py b/research/p104-findings-nov15/code/tails/build.py new file mode 100644 index 0000000000..211edd6170 --- /dev/null +++ b/research/p104-findings-nov15/code/tails/build.py @@ -0,0 +1,64 @@ +from lib import * +from chart_fn import clean,okpath,newpath +import copy +def apply(W,M,moves): + """moves: list of (event idx i, F'). Returns new word. Moves must touch disjoint stays.""" + paths={} + for i,F in moves: + for (r,k) in M.opnode[i]: + p=newpath(M,r,k,i,F) if r not in paths else None + assert p is not None,(i,r) + paths[r]=p + prog=W['program']; drop=set(); ins=collections.defaultdict(list) + for r,out in paths.items(): + nd=M.nodes[r] + for x in nd: + if x['mv'] is not None: drop.add(x['mv']) + # map chart-node -> first op position: walk original nodes, charts out; assign hops + # first op position per original stay in order; new nodes inherit via chart sequence + op ordering + evs=sorted(i for x in nd for i in x['ops']) + # recompute which chart each op sits at in the new path: ops in M.opnode order; ops belong to node (old); new chart for op = F if op in moves else old node chart + newc={} + for x in nd: + for i in x['ops']: newc[i]=x['c'] + for i,F in moves: + if any(rr==r for rr,_ in M.opnode[i]): newc[i]=F + # path positions: place hop into chart c' before first op at chart c' (in program order) if c' has ops else at end + seq=[newc[i] for i in evs] + first={} + for i,c in zip(evs,seq): first.setdefault(c,i) + # consistency: seq must follow out ordering + comp=clean([out[0]]+seq+[out[-1]]) + assert comp==out,(r,comp,out) + for a,b in zip(out,out[1:]): + pos=first.get(b) + if pos is None: pos=len(prog) + ins[pos].append(['MOVE',r,a,b,False]) + newprog=[] + mv={} + for i,e in enumerate(prog): + for m in ins.get(i,[]): newprog.append(m) + if i in drop: continue + e=list(e) + newprog.append(e) + for m in ins.get(len(prog),[]): newprog.append(m) + for i,F in moves: + pass + # rewrite ADD/SCALE chart tags + idx_new=[j for j,e in enumerate(newprog) if e[0]!='MOVE'] + old_nm=[i for i,e in enumerate(prog) if e[0]!='MOVE'] + assert len(idx_new)==len(old_nm) + for i,F in moves: + j=idx_new[old_nm.index(i)] if False else None + pos_map=dict(zip(old_nm,idx_new)) + for i,F in moves: + e=newprog[pos_map[i]] + e[4]=F + W2=dict(W); W2['program']=newprog; return W2 +if __name__=='__main__': + import sys + W=load('../BASE7612/w7612.json.gz'); M=Model(W) + mv=[(int(sys.argv[1]),int(sys.argv[2]))] + W2=apply(W,M,mv) + json.dump(W2,gzip.open(sys.argv[3],'wt')) + print(len(W['program']),len(W2['program'])) diff --git a/research/p104-findings-nov15/code/tails/census.py b/research/p104-findings-nov15/code/tails/census.py new file mode 100644 index 0000000000..4adb5cc64c --- /dev/null +++ b/research/p104-findings-nov15/code/tails/census.py @@ -0,0 +1,61 @@ +import os,sys,threading,resource,time,json,gzip,collections +os.environ['SUP_BASE']='inputs/ATTACK/COURIER226/hit_c226_1.json.gz' +def _w(): + while True: + if resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1e9>3.8: print('RSS>3.8GB bail',flush=True); os._exit(3) + time.sleep(1) +threading.Thread(target=_w,daemon=True).start() +sys.path.insert(0,'.') +from k3 import * +import circ +W8=json.load(gzip.open('inputs/ATTACK/PAIR/code/w7608.json.gz','rt')); nb8=circ.Base(W8) +def mapst(st,old): + h,F,a,d=st; L=old.stays[h]; i=[x[1] for x in L].index(a) + pf=L[i-1][0]; nf=L[i+1][0]; N=nb.stays[h] + c=[j for j,x in enumerate(N) if 0 (prev_frame or None, [stays in S], next_frame or None) + for r in wires: + L=B.stays[r]; idx=[i for i,st in enumerate(L) if st[0] in s.S] + prev=L[idx[0]-1][0] if idx[0]>0 else None + nxt=L[idx[-1]+1][0] if idx[-1]+11 else 244 + Wd=Window(B,B.downset(G)) + print('G',G,'wires',Wd.nw,'owned in window',len(Wd.owned),'gates',len(Wd.gates),'window loss route-sum %.3f'%Wd.loss()) + print('T nnz',sum(len(r) for r in Wd.T),'rank',rank_mod(list(range(Wd.nw)),list(range(Wd.nw)),Wd.T)) diff --git a/research/p104-findings-nov15/code/tails/dtimes.py b/research/p104-findings-nov15/code/tails/dtimes.py new file mode 100644 index 0000000000..86ccaf9e93 --- /dev/null +++ b/research/p104-findings-nov15/code/tails/dtimes.py @@ -0,0 +1,21 @@ +import os,sys,json +os.environ['SUP_BASE']='inputs/ATTACK/COURIER226/hit_c226_1.json.gz' +sys.path.insert(0,'.') +from census import * +from layer2 import delta_times +import tails +for st in [(680,1049,39784,43434),(592,1053,40278,42506),(681,1049,36664,43436),(664,1051,39828,43374),(117,1066,42259,43682),(627,117,33141,43993)]: + m=mapst(st,nbo); r=test([m],frames_mode='down',K=4000,owned=False,want=True) + ed,W2,org,seeds,sv=make([m]); r['org']=org; W3m,org2=corrected(r) + dt,Xn,Xb=delta_times(W3m,org2,B0) + # map W3 time -> base time approx via org2 + def bt(j): + while j0 else 0.0 +class Model: + def __init__(s,W): + s.W=W; n=W['public']; s.n=n + s.sp=[span(b) for b in W['frames']]; s.dim=[len(b) for b in W['frames']] + s.nodes=collections.defaultdict(list) # reg -> list of node dict(chart, ops, hop_ev) + for r in range(n): s.nodes[r].append(dict(c=W['initial'][r],ops=[],mv=None)) + s.owned_touch=set() + s.opnode={} # event idx -> list of (reg,node) + for i,e in enumerate(W['program']): + op,r=e[0],e[1] + if op=='MOVE': + if r>=n: continue + nd=s.nodes[r]; assert nd[-1]['c']==e[2]; nd.append(dict(c=e[3],ops=[],mv=i)) + elif op in('INIT','CLEAR'): pass + else: + regs=[r]+([e[2]] if op in('ADD','COPY') else []) + s.opnode[i]=[(x,len(s.nodes[x])-1 if x(evalset(S,True) if S else K0): S.append(r) + return S,evalset(S,True) if S else K0 +S,k=greedy([r for _,r in cand]); print('A-strict joint',len(S),g(k)) +S,k=greedy(loose); print('A-loose joint',len(S),g(k),[tuple(r['key'][:2]) for r in S][:20]) +P=collections.Counter(nat['P']); [stayP(P,r['key']) for r in S]; print('A-loose same stays, drops only (no waypoint cost)',g(kap(P))) diff --git a/research/p104-findings-nov15/code/tails/settle.py b/research/p104-findings-nov15/code/tails/settle.py new file mode 100644 index 0000000000..8137d6a963 --- /dev/null +++ b/research/p104-findings-nov15/code/tails/settle.py @@ -0,0 +1,52 @@ +"""TOKDEBT step 2: settlement rank of a clean-mode repaired schedule. Premix = invertible block-diagonal C0 (blocks = registers +sharing an initial frame, applied at t=0). Need Xnew_obs[:,R_f] C0_f = Xbase_obs[:,R_f] for every birth frame f. +rank = sum_f rank([Xn_f|Xb_f]) - rank(Xn_f) (mod p; 0 <=> solvable).""" +import sys, collections, numpy as np; sys.path.insert(0,'.'); sys.path.insert(0,'inputs/SEEDBUILD'); sys.path.insert(0,'inputs/NETCOMPILE') +import eng_c +from ort_c import * +from flint import nmod_mat +def tomat(A): return nmod_mat(A.shape[0], A.shape[1], [int(x) for x in A.ravel()], P) +def obs_rows(W2, org, Xb_obs_src): + """stack (new rows, base rows, kind) for public finals + surviving owned observations.""" + n2 = W2['public']; X, _, ow, cl = replay_mod(W2) + Xb, _, owb, clb = BREP + new = [X[r][:n2] for r in range(n2)]; base = [Xb[r][:n] for r in range(n)]; kind = [('pub', r) for r in range(n2)] + for j, row in ow.items(): + oi = org[j] + if oi is not None and oi in owb: new.append(row); base.append(owb[oi]); kind.append(('ow', j)) + for j, row in cl.items(): + oi = org[j] + if oi is not None and oi in clb: new.append(row); base.append(clb[oi]); kind.append(('cl', j)) + return np.array(new) % P, np.array(base) % P, kind +def corrected(r): + """W2 with corrections inserted (mod-p integer coefficients).""" + W2 = r['W2']; evs = collections.defaultdict(list) + for (a, b, pos, F), v in zip(r.get('cands', []), r.get('x', [])): + if v: evs[pos].append(['ADD', a, b, str(int(v)), F]) + prog = [] + org = list(r['org']) + org2 = [] + for j, e in enumerate(W2['program'] + [None]): + for ev in evs.get(j, []): prog.append(ev); org2.append(None) + if e is None: break + prog.append(e); org2.append(org[j]) + W3 = dict(W2); W3['program'] = prog + return W3, org2 +def settle(W3, org2, tier='A'): + mains = mains_of(B0); hc = [c for c in range(n) if c not in mains]; pc = sorted(mains) + Xn, Xb, kind = obs_rows(W3, org2, None) + Vh = {r for r in range(n) if B0['final'][r] == VFR and r not in mains} + keep = [i for i, k in enumerate(kind) if not (tier == 'A' and k[0] == 'pub' and k[1] in Vh)] + Xn = Xn[keep]; Xb = Xb[keep] + # pi check + pi_ok = not ((Xn[:, pc] - Xb[:, pc]) % P).any() if tier == 'A' else None + # birth frames + fr = collections.defaultdict(list) + for r in range(n): fr[B0['initial'][r]].append(r) + tot = 0; per = {} + for f_, R in fr.items(): + An = Xn[:, R]; Ab = Xb[:, R] + if not (An != Ab).any(): continue + rn = tomat(An).rank(); rj = tomat(np.hstack([An, Ab])).rank() + if rj > rn: per[f_] = rj - rn; tot += rj - rn + return tot, per, pi_ok, len(keep) diff --git a/research/p104-findings-nov15/code/tails/tab.py b/research/p104-findings-nov15/code/tails/tab.py new file mode 100644 index 0000000000..2832ab8727 --- /dev/null +++ b/research/p104-findings-nov15/code/tails/tab.py @@ -0,0 +1,25 @@ +import sys,json,collections +sys.path.insert(0,'.') +from tails import * +fin=W['final'] +def analyse(o,verbose=False): + rows=o.get('dirty',[]) + mains=[d for d in rows if d['kind']=='pub' and d['G']!=127] + v127=[d for d in rows if d['kind']=='pub' and d['G']==127] + own=[d for d in rows if d['kind']!='pub'] + res=dict(nmain=len(mains),nv=len(v127),nown=len(own),need=collections.Counter(),pairs=set(),colsnonhelper=0) + for d in mains: + r,G=d['r'],d['G']; Tm=arrive(r,G) + for c in d['cols']: + if fin[c]!=127: res['colsnonhelper']+=1; st=('col-main',None) + else: st=opt(c,G,Tm) + res['need'][st[0]]+=1 + if st[0]=='ok': res['pairs'].add((c,G)) + if verbose: I=info(c); print(' main',r,'G',G,'dG',M.dim[G],'T',Tm,'| helper',c,'L',I['L'],'dL',I['dL'],'F',I['F'],'lastop',I['lastop'],'firstF',I['firstF'],'->',st) + return res +if __name__=='__main__': + for l in open(sys.argv[1]): + o=json.loads(l); + if 'dirty' not in o: print(o['st'],o.get('err')); continue + print(o['st'],'ok',o['ok']); a=analyse(o,verbose=len(sys.argv)>2) + print(' mains',a['nmain'],'V127',a['nv'],'owned',a['nown'],dict(a['need'])) diff --git a/research/p104-findings-nov15/code/tails/tails.py b/research/p104-findings-nov15/code/tails/tails.py new file mode 100644 index 0000000000..c7a65fd1e6 --- /dev/null +++ b/research/p104-findings-nov15/code/tails/tails.py @@ -0,0 +1,36 @@ +import sys,json,collections,math +sys.path.insert(0,'.') +from lib import load,Model,span +W=load('inputs/ATTACK/COURIER226/hit_c226_1.json.gz'); M=Model(W); n=W['public'] +K=0.00365455370060 +def f(d): return d**(1-K) if d>0 else 0.0 +def sub(a,b): return M.sp[a]&~M.sp[b]==0 # a subset-or-equal b +def ssub(a,b): return sub(a,b) and M.sp[a]!=M.sp[b] +prog=W['program'] +def info(h): + nd=M.nodes[h] + w=[k for k in range(len(nd)-1) if nd[k]['ops']] + k=w[-1] if w else 0 + L=nd[k]['c']; lastop=nd[k]['ops'][-1] if nd[k]['ops'] else (nd[k]['mv'] or 0) + fin=nd[-1]; firstF=fin['ops'][0] if fin['ops'] else len(prog) + # does h get written (dst) at its final node? + dstF=sum(1 for i in fin['ops'] if prog[i][1]==h) + tail=[x['c'] for x in nd[k:]] + return dict(L=L,dL=M.dim[L],k=k,lastop=lastop,F=fin['c'],firstF=firstF,nF=len(fin['ops']),dstF=dstF,tail=tail,tailcost=sum(f(abs(M.dim[b]-M.dim[a])) for a,b in zip(tail,tail[1:]))) +def arrive(r,G): + nd=M.nodes[r] + for x in nd: + if x['c']==G: return x['mv'] if x['mv'] is not None else 0 + return None +def opt(h,G,Tm): + """can helper h be at chart G (same frame) after main arrival Tm, inside its post-work window? returns (status, dcost)""" + I=info(h) + if I['F']!=127 and not (I['F']==G): return ('notV',None) + if not sub(I['L'],G): return ('L-not-under-G',None) + if not ssub(G,I['F']): return ('G-not-under-F',None) + if I['firstF']<=Tm: return ('time',None) + new=[I['L']]+([G] if I['L']!=G else [])+[I['F']] + dc=sum(f(abs(M.dim[b]-M.dim[a])) for a,b in zip(new,new[1:]))-I['tailcost'] + return ('ok' if I['L']!=G else 'ok-atG',dc) +if __name__=='__main__': + for h in map(int,sys.argv[1:]): print(h,info(h)) diff --git a/research/p104-findings-nov15/data/7113_CALLER.json.gz b/research/p104-findings-nov15/data/7113_CALLER.json.gz new file mode 100644 index 0000000000..8a6cac5f96 Binary files /dev/null and b/research/p104-findings-nov15/data/7113_CALLER.json.gz differ diff --git a/research/p104-findings-nov15/data/w7546.json.gz b/research/p104-findings-nov15/data/w7546.json.gz new file mode 100644 index 0000000000..d63f6b195e Binary files /dev/null and b/research/p104-findings-nov15/data/w7546.json.gz differ