Embedded Language Support + LiveDoc (#30)

* allows clojure mode to operate when embedded in other languages (e.g. Markdown)
* first sketch of a cljs Markdown notebook editor with eval support in Clerk's SCI context

Author: zampino

.github/workflows/main.yml

@@ -11,7 +11,7 @@ jobs:
           java-version: '11.0.7'
 
       - name: 🔧 Install clojure
-        uses: DeLaGuardo/setup-clojure@master
+        uses: DeLaGuardo/setup-clojure@9.0
         with:
           cli: '1.10.3.943'
 
@@ -64,12 +64,22 @@ jobs:
           branch: gh-pages     # The branch the action should deploy to.
           folder: public       # The folder the action should deploy.
 
-      - name: ✅ Add success status to report with link to snapshot
+      - name: ✅ Add link to Clojure Mode Demo
         uses: Sibz/github-status-action@v1
         with:
           authToken: ${{secrets.GITHUB_TOKEN}}
-          context: 'CI / Static App / URL'
+          context: 'CI / Static App / Clojure Mode'
           description: 'Ready'
           state: 'success'
           sha: ${{github.event.pull_request.head.sha || github.sha}}
           target_url: https://snapshots.nextjournal.com/clojure-mode/build/${{ github.sha }}
+
+      - name: ✅ Add link to LiveDoc Demo
+        uses: Sibz/github-status-action@v1
+        with:
+          authToken: ${{secrets.GITHUB_TOKEN}}
+          context: 'CI / Static App / LiveDoc'
+          description: 'Ready'
+          state: 'success'
+          sha: ${{github.event.pull_request.head.sha || github.sha}}
+          target_url: https://snapshots.nextjournal.com/clojure-mode/build/${{ github.sha }}/livedoc

.gitignore

@@ -5,6 +5,7 @@
 .cpcache
 .lsp
 public/js
+public/livedoc/js
 node_modules
 public/test
 *.iml

bb.edn

@@ -0,0 +1,15 @@
+{:min-bb-version "0.7.6"
+ :paths ["bb"]
+ :tasks
+ {:requires ([clojure.edn :as edn]
+             [clojure.string :as str]
+             [babashka.deps :as deps]
+             [babashka.fs :as fs]
+             [babashka.process :as p])
+  :init (do
+          (defn viewer-css-path []
+            (let [cp (str/trim (with-out-str (deps/clojure ["-A:dev:demo" "-Spath"])))]
+              (str/trim (:out (shell {:out :string} (str "bb -cp " cp " -e '(println (.getPath (clojure.java.io/resource \"css/viewer.css\")))'")))))))
+
+  copy-viewer-css {:doc "Copies viewer stylesheet to resources."
+                   :task (fs/copy (viewer-css-path) "resources/stylesheets/viewer.css" #{:replace-existing})}}}

demo/deps.edn

@@ -0,0 +1,139 @@
+# 👋 Hello LiveDoc
+
+LiveDoc is a cljs notebook editor powered by [CodeMirror Markdown language support](https://github.com/codemirror/lang-markdown) and [nextjournal clojure mode](https://nextjournal.github.io/clojure-mode).
+
+In this demo we're evaluating code in [Clerk](https://github.com/nextjournal/clerk)'s SCI context. In particular we're rendering _markdown_ cells in terms of Clerk's viewers. This allows e.g. to get inline $\LaTeX$ formulas as well as block ones
+
+$$\hat{f}(x) = \int_{-\infty}^{+\infty} f(t)\exp^{-2\pi i x t}dt$$
+
+Here's some of Clerk's API in action
+
+```clojure
+(v/vl {:width 650 :height 400 :mark "geoshape"
+       :data {:url "https://vega.github.io/vega-datasets/data/us-10m.json"
+              :format {:type "topojson"
+                       :feature "counties"}}
+       :transform
+       [{:lookup "id"
+         :from {:data {:url "https://vega.github.io/vega-datasets/data/unemployment.tsv"}
+                :key "id" :fields ["rate"]}}]
+       :projection {:type "albersUsa"}
+       :encoding {:color {:field "rate" :type "quantitative"}}})
+```
+
+Compose results layout with `v/row` and `v/col`
+
+```clojure
+(def pie
+  (v/plotly
+    {:data [{:values [27 11 25 8 1 3 25]
+             :labels ["US" "China" "European Union" "Russian Federation"
+                      "Brazil" "India" "Rest of World"]
+             :text "CO2"
+             :textposition "inside"
+             :domain {:column 1}
+             :hoverinfo "label+percent+name"
+             :hole 0.4
+             :type "pie"}]
+     :layout {:showlegend false
+              :width 200
+              :height 200
+              :annotations [{:font {:size 20} :showarrow false :x 0.5 :y 0.5 :text "CO2"}]}
+     :config {:responsive true}}))
+
+(def contour
+  (v/plotly {:data [{:z [[10 10.625 12.5 15.625 20]
+                         [5.625 6.25 8.125 11.25 15.625]
+                         [2.5 3.125 5.0 8.125 12.5]
+                         [0.625 1.25 3.125 6.25 10.625]
+                         [0 0.625 2.5 5.625 10]]
+                     :type "contour"}]}))
+
+(v/col
+  ;; FIXME: can't use nested v/html
+  (v/with-viewer :html [:h1 "Plots"])
+  (v/row pie contour))
+```
+
+## Extending the Evaluation Context
+
+The rendering of blocks and their evaluation is fully customizable, this makes it easy to bring your own SCI context. In this notebook Clerk's context is being augmented of some convenient helpers for loading and handling data in the notebook:
+
+* `livedoc/with-fetch`
+* `csv/parse`
+* `observable/Plot`
+
+```clojure
+(livedoc/with-fetch "https://gist.githubusercontent.com/netj/8836201/raw/6f9306ad21398ea43cba4f7d537619d0e07d5ae3/iris.csv"
+  (fn [text]
+    (v/table (take 10 (map js->clj (csv/parse text))))))
+```
+
+The following example is taken from this [Observable notebook](https://observablehq.com/@observablehq/plot)
+
+```clojure
+(livedoc/with-fetch "https://raw.githubusercontent.com/flother/rio2016/master/athletes.csv"
+  (fn [data]
+    (.. observable/Plot
+      (dot (csv/parse data)
+        (j/obj :x "weight" :y "height" :stroke "sex"))
+      plot)))
+```
+```clojure
+(v/plotly {:data [{:y (shuffle (range -100 100))}]})
+```
+
+## Usage
+
+Use livedoc `editor` function as a reagent component in your cljs application
+
+    [nextjournal.clojure-mode.livedoc/editor opts]
+
+this puts together an instance of CodeMirror with markdown and clojure mixed language support with a set of extensions configurable via an `opts` map with keys:
+
+* `:doc` (required) a markdown string
+
+* `:render` a function taking a reagent state atom, returning hiccup. Such state holds a map with:
+    * `:text` the block's text
+    * `:type` with values `:code` or `:markdown`
+    * `:selected?`
+
+* `:eval-fn!` will be called on selected block states when evaluation is triggered
+
+* `:tooltip` customises tooltip view
+
+* `:extensions` extra CodeMirror extensions to be added along livedoc ones
+
+* `:focus?` should editor acquire focus when loaded
+
+## Keybindings
+
+* `ESC`: toggles edit-one / edit-all / preview & select block
+* `ALT`: pressed while in edit mode toggles a tooltip with eval-at-cursor results
+* Arrow keys move selection up/down
+* `CMD + Enter` : Evaluate selected cell or leave edit mode
+* `CMD + Shift + Enter`: Evaluates all cells
+
+```clojure
+(defonce state (atom 0))
+```
+```clojure
+(defn the-answer
+  "to all questions"
+  [x]
+  (inc x))
+```
+```clojure
+(swap! state inc)
+```
+```clojure
+(v/html [:h2 (str "The Answer is: " (the-answer @state))])
+```
+
+## Todo
+- [ ] cannot click to move cursor in each editable section bottom lines (probably we need calls to `requestMeasure`) 
+- [ ] scroll selected block into view when moving out of viewport
+- [ ] clicking on blocks not always results in an edit at the right place
+- [ ] avoid re-rendering _all_ previews when scrolling or clicking to edit one (probably connected to height computations)
+- [ ] use async SCI eval
+- [ ] don't eval code when rendering previews (but only when leaving edit mode)

demo/notebooks/livedoc.md

@@ -1,3 +1,4 @@
 {:npm-deps
  {"react" "^17.0.2"
-  "react-dom" "^17.0.2"}}
+  "react-dom" "^17.0.2"
+  "framer-motion" "^6.2.8"}}

demo/src/deps.cljs

@@ -4,23 +4,27 @@
             ["@codemirror/state" :refer [EditorState]]
             ["@codemirror/view" :as view :refer [EditorView]]
             [nextjournal.clerk.sci-viewer :as sv]
-            [nextjournal.clerk.viewer :as clerk.viewer]
+            [nextjournal.clerk.viewer :as v]
             [applied-science.js-interop :as j]
+            [shadow.resource :as rc]
             [clojure.string :as str]
             [nextjournal.clojure-mode :as cm-clj]
-            [nextjournal.clojure-mode.demo.sci :as sci]
+            [nextjournal.livedoc :as livedoc]
+            [nextjournal.clojure-mode.demo.sci :as demo.sci]
             [nextjournal.clojure-mode.keymap :as keymap]
             [nextjournal.clojure-mode.live-grammar :as live-grammar]
             [nextjournal.clojure-mode.test-utils :as test-utils]
-            [react]
+            ["react" :as react]
             [reagent.core :as r]
             [reagent.dom :as rdom]))
 
 (def theme
   (.theme EditorView
           (j/lit {".cm-content" {:white-space "pre-wrap"
-                                 :padding "10px 0"}
-                  "&.cm-focused" {:outline "none"}
+                                 :padding "10px 0"
+                                 :flex "1 1 0"}
+
+                  "&.cm-focused" {:outline "0 !important"}
                   ".cm-line" {:padding "0 9px"
                               :line-height "1.6"
                               :font-size "16px"
@@ -51,15 +55,15 @@
 
 (defn editor [source {:keys [eval?]}]
   (r/with-let [!view (r/atom nil)
-               last-result (when eval? (r/atom (sci/eval-string source)))
+               last-result (when eval? (r/atom (demo.sci/eval-string source)))
                mount! (fn [el]
                         (when el
                           (reset! !view (new EditorView
                                              (j/obj :state
                                                     (test-utils/make-state
                                                      (cond-> #js [extensions]
-                                                       eval? (.concat #js [(sci/extension {:modifier  "Alt"
-                                                                                           :on-result (partial reset! last-result)})]))
+                                                       eval? (.concat #js [(demo.sci/extension {:modifier "Alt"
+                                                                                                :on-result (partial reset! last-result)})]))
                                                      source)
                                                     :parent el)))))]
     [:div
@@ -74,8 +78,27 @@
             (react/isValidElement result) result
             'else (sv/inspect-paginated result)))])]
     (finally
-      (j/call @!view :destroy))))
-
+     (j/call @!view :destroy))))
+
+;;  Markdown editors
+(defn markdown-editor [{:keys [doc extensions]}]
+  [:div {:ref (fn [^js el]
+                (when el
+                  (some-> el .-editorView .destroy)
+                  (j/assoc! el :editorView
+                            (EditorView. (j/obj :parent el
+                                                :state (.create EditorState
+                                                                (j/obj :doc (str/trim doc)
+                                                                       :extensions (into-array
+                                                                                    (cond-> [(syntaxHighlighting defaultHighlightStyle)
+                                                                                             (foldGutter)
+                                                                                             (.of view/keymap cm-clj/complete-keymap)
+                                                                                             (history)
+                                                                                             (.of view/keymap historyKeymap)
+                                                                                             theme
+                                                                                             livedoc/markdown-language-support]
+                                                                                      (seq extensions)
+                                                                                      (concat extensions))))))))))}])
 
 (defn samples []
   (into [:<>]
@@ -123,7 +146,7 @@
                           (when-not (zero? i) [:span " + "])
                           [:kbd.kbd k]]) keys))))
 
-(defn key-bindings-table []
+(defn key-bindings-table [keymap]
   [:table.w-full.text-sm
    [:thead
     [:tr.border-t
@@ -132,8 +155,7 @@
      [:th.px-3.py-1.align-top.text-left.text-xs.uppercase.font-normal.black-50 "Alternate Binding"]
      [:th.px-3.py-1.align-top.text-left.text-xs.uppercase.font-normal.black-50 {:style {:min-width 290}} "Description"]]]
    (into [:tbody]
-         (->> keymap/paredit-keymap*
-              (merge (sci/keymap* "Alt"))
+         (->> keymap
               (sort-by first)
               (map (fn [[command [{:keys [key shift doc]} & [{alternate-key :key}]]]]
                      [:<>
@@ -152,17 +174,59 @@
 
 (defn ^:dev/after-load render []
   (rdom/render [samples] (js/document.getElementById "editor"))
-
   (.. (js/document.querySelectorAll "[clojure-mode]")
       (forEach #(when-not (.-firstElementChild %)
                   (rdom/render [editor (str/trim (.-innerHTML %))] %))))
-
   (let [mapping (key-mapping)]
     (.. (js/document.querySelectorAll ".mod,.alt,.ctrl")
         (forEach #(when-let [k (get mapping (.-innerHTML %))]
                     (set! (.-innerHTML %) k)))))
 
-  (rdom/render [key-bindings-table] (js/document.getElementById "docs"))
+  ;; set viewer tailwind stylesheet
+  (j/assoc! (js/document.getElementById "viewer-stylesheet")
+            :innerHTML (rc/inline "stylesheets/viewer.css"))
+
+  (rdom/render [key-bindings-table (merge keymap/paredit-keymap* (demo.sci/keymap* "Alt"))] (js/document.getElementById "docs"))
+  (rdom/render [:div.rounded-md.mb-0.text-sm.monospace.overflow-auto.relative.border.shadow-lg.bg-white
+                [markdown-editor {:doc "# Hello Markdown
+
+Lezer [mounted trees](https://lezer.codemirror.net/docs/ref/#common.MountedTree) allows to
+have an editor with ~~mono~~ _mixed language support_.
+
+```clojure
+(defn the-answer
+  \"to all questions\"
+  []
+  (inc 41))
+```
+
+## Todo
+- [x] resolve **inner nodes**
+- [x] fix extra spacing when autoformatting after paredit movements
+- [x] fix errors when entering a newline
+- [ ] fix extra space when entering a newline
+- [x] fix nonsense deletions hitting delete key
+- [x] limit the scope of autoformat (TAB)
+- [x] limit the scope of kill*
+- [x] limit the scope of eval-region
+- [ ] restore autoformat when deleting
+- [x] keep parens balanced when deleting backward
+- [x] fix errors on Ctrl-K
+- [ ] fix dark theme
+- [ ] fix demo error: CssSyntaxError: <css input>:62:15: The `font-inter` class does not exist
+"}]] (js/document.getElementById "markdown-editor"))
 
   (when (linux?)
     (js/twemoji.parse (.-body js/document))))
+
+(comment
+  (let [ctx' (sci.core/fork @sv/!sci-ctx)
+        ctx'' (sci.core/merge-opts ctx' {:namespaces {'foo {'bar "ahoi"}}})]
+
+    (demo.sci/eval-string ctx'' "(def o (j/assoc! #js {:a 1} :b 2))")
+    (demo.sci/eval-string ctx'' "(j/lookup (j/assoc! #js {:a 1} :b 2))")
+    (demo.sci/eval-string ctx'' "(j/get o :b)")
+    (demo.sci/eval-string ctx'' "(into-array [1 2 3])")
+
+    ;; this is not evaluable as-is in sci
+    (demo.sci/eval-string ctx'' "(j/let [^:js {:keys [a b]} o] (map inc [a b]))")))