Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

helm-org-in-buffer-headings is very slow in big org-mode file #1964

Closed
stardiviner opened this issue Feb 6, 2018 · 10 comments
Closed

helm-org-in-buffer-headings is very slow in big org-mode file #1964

stardiviner opened this issue Feb 6, 2018 · 10 comments

Comments

@stardiviner
Copy link

stardiviner commented Feb 6, 2018

Describe versions of Helm, Emacs, operating system, etc.

  • Emacs version: GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.26) of 2018-01-31
  • Helm version: MELPA
  helm               20180205.1023 dependency            Helm is an Emacs incremental and narrowing framework
  helm-core          20180203.759  dependency            Development files for Helm

I use Emacs built-in profiler to profile. Bellowing is the result.

Table of Contents

  1. profiler result
    1. Memory
    2. CPU

profiler result

Memory

- command-execute                                          70,070,223  95%
 - apply                                                   70,070,223  95%
  - ad-Advice-command-execute                              70,070,223  95%
   - #<compiled 0x24d23b>                                  70,070,223  95%
    - call-interactively                                   70,070,223  95%
     - funcall-interactively                               70,070,223  95%
      - helm-org-in-buffer-headings                        64,303,905  88%
       - let                                               64,303,905  88%
        - helm                                             64,303,905  88%
         - apply                                           64,149,585  87%
          - helm                                           64,149,585  87%
           - apply                                         64,149,585  87%
            - helm-internal                                64,149,585  87%
             - helm-read-pattern-maybe                     61,117,310  83%
              - helm-update                                57,442,027  78%
               - helm--collect-matches                     47,369,597  64%
                - helm-compute-matches                     47,369,597  64%
                 - helm-process-filtered-candidate-transformer    47,369,597  64%
                  - helm-funcall-with-source               47,369,597  64%
                   - apply                                 47,369,597  64%
                    - helm-org-startup-visibility          47,369,597  64%
                     - let*                                47,369,597  64%
                      - while                              47,369,597  64%
                       - setq                              47,369,597  64%
                        - cons                             47,363,261  64%
                         - if                              47,363,261  64%
                          - cons                           47,355,869  64%
                           - if                            47,355,869  64%
                            - if                           47,355,869  64%
                             - progn                       47,231,965  64%
                              - let                        47,225,629  64%
                               - save-current-buffer       47,072,509  64%
                                - cond                     47,072,509  64%
                                 - save-current-buffer     47,072,509  64%
                                  - org-indent-mode        44,981,537  61%
                                   - let                   44,967,809  61%
                                    - cond                 44,790,369  61%
                                     - org-indent--compute-prefixes    44,515,273  60%
                                      - let                44,347,017  60%
                                       - while             44,347,017  60%
                                        - let              43,742,985  59%
                                         - let             41,286,689  56%
                                          - aset           23,630,218  32%
                                           - concat        13,665,962  18%
                                              org-add-props     7,820,802  10%
                                              and           1,531,156   2%
                                          + let            11,507,718  15%
                                        setq                  132,352   0%
                                     + and                    245,528   0%
                                       add-hook                 6,336   0%
                                       org-indent-remove-properties         6,336   0%
                                     + if                       3,168   0%
                                    + if                      111,936   0%
                                  + replace-match             742,820   1%
                               + org-find-invisible-foreground       153,120   0%
               - helm-render-source                         8,381,865  11%
                - helm-insert-match                         8,378,579  11%
                 - put-text-property                        5,659,684   7%
                  - org-indent-refresh-maybe                4,189,732   5%
                   - if                                     4,189,732   5%
                    - progn                                 4,189,732   5%
                     - let                                  4,135,876   5%
                      - unwind-protect                      3,917,284   5%
                       - progn                              3,917,284   5%
                        - save-excursion                    3,917,284   5%
                         - save-restriction                 3,917,284   5%
                          - if                              3,917,284   5%
                           - let                            3,863,428   5%
                            - org-indent-add-properties     3,285,912   4%
                             - let                          3,231,000   4%
                              - unwind-protect              3,011,352   4%
                               - progn                      3,011,352   4%
                                - save-excursion            2,748,408   3%
                                 - save-restriction         2,748,408   3%
                                  - let*                    2,460,120   3%
                                   - or                     1,286,904   1%
                                    - org-current-level     1,286,904   1%
                                     - let                  1,205,592   1%
                                      - progn               1,205,592   1%
                                       - let*               1,205,592   1%
                                        + org-outline-level       910,336   1%
                                        + org-get-limited-outline-regexp        82,368   0%
                                   + let*                   1,009,536   1%
                                   + org-get-limited-outline-regexp        82,368   0%
                            + save-excursion                  412,780   0%
                  + org-indent-notify-modified-headline     1,142,592   1%
                 + insert                                   2,281,711   3%
                + helm-insert-header-from-source                3,286   0%
               + #<compiled 0x3f4c3cd>                      1,569,333   2%
               + helm--update-move-first-line                 120,208   0%
                 helm-update-source-p                           1,024   0%
              + read-from-minibuffer                        3,580,841   4%
              + thing-at-point                                  1,024   0%
             + helm-display-buffer                          1,540,673   2%
             + #<compiled 0x193c7f9>                        1,479,930   2%
             + helm-execute-selection-action                    6,480   0%
               advice-add                                       4,136   0%
               #<compiled 0x2ee2d35>                            1,056   0%
         + helm-source-org-headings-for-files                 153,296   0%
           helm-parse-keys                                      1,024   0%
      + profiler-report                                     5,766,318   7%
- flyspell-post-command-hook                                2,276,410   3%
 - flyspell-word                                            2,276,410   3%
  - org-mode-flyspell-verify                                2,268,026   3%
   - if                                                     2,268,026   3%
    - let*                                                  2,267,002   3%
     - cond                                                 2,240,642   3%
      - let                                                 2,240,642   3%
       - org-log-into-drawer                                2,240,642   3%
        - let                                               2,240,642   3%
         - org-entry-get                                    2,240,642   3%
          - let                                             2,240,642   3%
           - save-excursion                                 2,240,642   3%
            + save-excursion                                2,240,642   3%
     + org-element-at-point                                    26,360   0%
    + org-at-heading-p                                          1,024   0%
  + flyspell-get-word                                           8,384   0%
+ ...                                                         304,940   0%
+ redisplay_internal (C function)                             296,270   0%
+ timer-event-handler                                          49,366   0%
+ yas--post-command-handler                                    28,704   0%
  beacon--vanish                                                2,048   0%
+ global-font-lock-mode-check-buffers                           1,056   0%
+ global-edit-server-edit-mode-check-buffers                    1,056   0%
+ yas-global-mode-check-buffers                                 1,056   0%
+ winner-save-old-configurations                                1,056   0%
+ rtags-post-command-hook                                       1,056   0%
+ mode-local-post-major-mode-change                             1,056   0%

CPU

- command-execute                                                5849  73%
 - apply                                                         5849  73%
  - ad-Advice-command-execute                                    5849  73%
   - #<compiled 0x24d23b>                                        5849  73%
    - call-interactively                                         5849  73%
     - funcall-interactively                                     5849  73%
      - helm-org-in-buffer-headings                              5833  73%
       - let                                                     5833  73%
        - helm                                                   5833  73%
         - apply                                                 5826  72%
          - helm                                                 5826  72%
           - apply                                               5826  72%
            - helm-internal                                      5826  72%
             - helm-read-pattern-maybe                           5790  72%
              - helm-update                                      5109  63%
               - helm--collect-matches                           2731  34%
                - helm-compute-matches                           2731  34%
                 - helm-process-filtered-candidate-transformer               2731  34%
                  - helm-funcall-with-source                     2731  34%
                   - apply                                       2731  34%
                    - helm-org-startup-visibility                2731  34%
                     - let*                                      2731  34%
                      - while                                    2731  34%
                       - setq                                    2731  34%
                        - cons                                   2731  34%
                         - if                                    2731  34%
                          - cons                                 2724  34%
                           - if                                  2720  34%
                            - if                                 2720  34%
                             - progn                             2700  33%
                              - let                              2696  33%
                               - save-current-buffer               2408  30%
                                + cond                           2381  29%
                                - set-buffer                        3   0%
                                 - or                               3   0%
                                    and                             3   0%
                               - org-find-invisible-foreground                276   3%
                                - let                             276   3%
                                 - remove                         276   3%
                                  - nconc                         273   3%
                                   - list                         260   3%
                                    - face-background                207   2%
                                       face-attribute                204   2%
                                    - face-foreground                 50   0%
                                       face-attribute                 50   0%
                                   - mapcar                        10   0%
                                      #<lambda 0x12fb837447e63502>                  4   0%
               - helm-render-source                              2365  29%
                - helm-insert-match                              2341  29%
                 + put-text-property                             1373  17%
                 - insert                                         811  10%
                  - org-indent-notify-modified-headline                485   6%
                   - if                                           485   6%
                    - progn                                       485   6%
                     - setq                                       485   6%
                      - save-excursion                            478   5%
                       + save-restriction                         478   5%
                  + org-indent-refresh-maybe                      261   3%
                  + beacon--vanish                                 39   0%
               + #<compiled 0x3f4c3cd>                             10   0%
               + helm--update-move-first-line                       3   0%
              + read-from-minibuffer                              632   7%
                helm-get-candidate-number                          10   0%
             + helm-display-buffer                                 14   0%
             + #<compiled 0x193c7f9>                               10   0%
             + helm-initialize                                      7   0%
               #<compiled 0x2ee2d35>                                5   0%
         + helm-source-org-headings-for-files                       7   0%
        profiler-start                                             11   0%
      + profiler-report                                             5   0%
- ...                                                            2078  26%
   Automatic GC                                                  1745  21%
 + helm-transform-candidates                                      333   4%
+ timer-event-handler                                              31   0%
+ redisplay_internal (C function)                                  20   0%
  global-anzu-mode-check-buffers                                    3   0%
+ flyspell-post-command-hook                                        3   0%
@thierryvolpiatto
Copy link
Member

thierryvolpiatto commented Feb 6, 2018 via email

@stardiviner
Copy link
Author

I see, Thanks @thierryvolpiatto

@matthuszagh
Copy link

Would it be possible to perhaps cache the results in a file and then use the cache for faster retrieval? The cache could be generated asynchronously with another command (that in turn could be tied to save file hook, or whatever the user wants). Since the cache could become outdated this probably shouldn't be the default behavior, but it could be a useful customization option when dealing with large org buffers.

@thierryvolpiatto
Copy link
Member

thierryvolpiatto commented Nov 28, 2019 via email

@thierryvolpiatto
Copy link
Member

Can you try if after applying this patch it is better?

diff --git a/helm-org.el b/helm-org.el
index ab8e220a..662a4002 100644
--- a/helm-org.el
+++ b/helm-org.el
@@ -202,88 +202,25 @@ Note: [1] A separator can be a comma, a colon i.e. [,:] or a space.
     map)
   "Keymap for `helm-source-org-headings-for-files'.")
 
-(defclass helm-org-headings-class (helm-source-sync)
-  ((parents
-    :initarg :parents
-    :initform nil
-    :custom boolean)
-   (match :initform
-          (lambda (candidate)
-            (string-match
-             helm-pattern
-             (helm-aif (get-text-property 0 'helm-real-display candidate)
-                 it
-               candidate))))
-   (help-message :initform 'helm-org-headings-help-message)
-   (action :initform 'helm-org-headings-actions)
-   (keymap :initform 'helm-org-headings-map)
-   (group :initform 'helm-org)))
-
-(defmethod helm--setup-source :after ((source helm-org-headings-class))
-  (let ((parents (slot-value source 'parents)))
-    (setf (slot-value source 'candidate-transformer)
-          (lambda (candidates)
-            (let ((cands (helm-org-get-candidates candidates parents)))
-              (if parents (nreverse cands) cands))))))
-
 (defun helm-source-org-headings-for-files (filenames &optional parents)
   "Build source for org headings in files FILENAMES.
 When PARENTS is specified, bild source for heading that are parents of
 current heading."
-  (helm-make-source "Org Headings" 'helm-org-headings-class
-    :filtered-candidate-transformer 'helm-org-startup-visibility
-    :parents parents
-    :candidates filenames))
-
-(defun helm-org-startup-visibility (candidates _source)
-  "Indent headings and hide leading stars displayed in the helm buffer.
-If `org-startup-indented' and `org-hide-leading-stars' are nil, do
-nothing to CANDIDATES."
-  (cl-loop for i in candidates
-	   collect
-           ;; Transformation is not needed if these variables are t.
-	   (if (or helm-org-show-filename helm-org-format-outline-path)
-	       (cons
-		(car i) (cdr i))
-             (cons
-              (if helm-org-headings-fontify
-                  (when (string-match "^\\(\\**\\)\\(\\* \\)\\(.*\n?\\)" (car i))
-                    (replace-match "\\1\\2\\3" t nil (car i)))
-                (when (string-match "^\\(\\**\\)\\(\\* \\)\\(.*\n?\\)" (car i))
-                  (let ((foreground (org-find-invisible-foreground)))
-                    (with-helm-current-buffer
-                      (cond
-                       ;; org-startup-indented is t, and org-hide-leading-stars is t
-                       ;; Or: #+STARTUP: indent hidestars
-                       ((and org-startup-indented org-hide-leading-stars)
-                        (with-helm-buffer
-                          (require 'org-indent)
-                          (org-indent-mode 1)
-                          (replace-match
-                           (format "%s\\2\\3"
-                                   (propertize (replace-match "\\1" t nil (car i))
-                                               'face `(:foreground ,foreground)))
-                           t nil (car i))))
-                       ;; org-startup-indented is nil, org-hide-leading-stars is t
-                       ;; Or: #+STARTUP: noindent hidestars
-                       ((and (not org-startup-indented) org-hide-leading-stars)
-                        (with-helm-buffer
-                          (replace-match
-                           (format "%s\\2\\3"
-                                   (propertize (replace-match "\\1" t nil (car i))
-                                               'face `(:foreground ,foreground)))
-                           t nil (car i))))
-                       ;; org-startup-indented is nil, and org-hide-leading-stars is nil
-                       ;; Or: #+STARTUP: noindent showstars
-                       (t
-                        (with-helm-buffer
-                          (replace-match "\\1\\2\\3" t nil (car i)))))))))
-              (cdr i)))))
+  (helm-build-sync-source "Org headings"
+    :candidates (helm-dynamic-completion
+                 (helm-org-get-candidates
+                  filenames parents)
+                 'stringp)
+    :match-dynamic t
+    :action 'helm-org-headings-actions
+    :keymap helm-org-headings-map
+    :group 'helm-org
+    :requires-pattern 2))
 
 (defun helm-org-get-candidates (filenames &optional parents)
   "Get org headings for file FILENAMES.
 Get PARENTS of heading when specified."
-  (apply #'append
+  (apply #'append ; Flatten list.
          (mapcar (lambda (filename)
                    (helm-org--get-candidates-in-file
                     filename
@@ -324,7 +261,11 @@ Get PARENTS as well when specified."
           (and (boundp 'org-outline-path-cache)
                (setq org-outline-path-cache nil))
           (cl-loop with width = (window-width (helm-window))
-                   while (funcall search-fn)
+                   while (and (funcall search-fn)
+                              (string-match-p
+                               helm-pattern
+                               (buffer-substring-no-properties
+                                (point-at-bol) (point-at-eol))))
                    for beg = (point-at-bol)
                    for end = (point-at-eol)
                    when (and fontify
@@ -418,12 +359,15 @@ will be refiled."
                                   (expand-file-name
                                    (concat "#" (helm-basename f) "#")
                                    (helm-basedir f)))
-                            collect (helm-basename f))))
+                            collect (helm-basename f)))
+        (files (org-agenda-files))
+        org-hide-leading-stars
+        org-startup-indented)
     (when (or (null autosaves)
               helm-org-ignore-autosaves
               (y-or-n-p (format "%s have auto save data, continue? "
                                 (mapconcat #'identity autosaves ", "))))
-      (helm :sources (helm-source-org-headings-for-files (org-agenda-files))
+      (helm :sources (helm-source-org-headings-for-files files)
             :candidate-number-limit 99999
             :truncate-lines helm-org-truncate-lines
             :buffer "*helm org headings*"))))
@@ -432,9 +376,11 @@ will be refiled."
 (defun helm-org-in-buffer-headings ()
   "Preconfigured helm for org buffer headings."
   (interactive)
-  (let (helm-org-show-filename)
-    (helm :sources (helm-source-org-headings-for-files
-                    (list (current-buffer)))
+  (let (helm-org-show-filename
+        (files (list (current-buffer)))
+        org-startup-indented
+        org-hide-leading-stars)
+    (helm :sources (helm-source-org-headings-for-files files)
           :candidate-number-limit 99999
           :preselect (helm-org-in-buffer-preselect)
           :truncate-lines helm-org-truncate-lines
@@ -446,9 +392,9 @@ will be refiled."
   (interactive)
   ;; Use a large max-depth to ensure all parents are displayed.
   (let ((helm-org-headings-min-depth 1)
-        (helm-org-headings-max-depth  50))
-    (helm :sources (helm-source-org-headings-for-files
-                    (list (current-buffer)) t)
+        (helm-org-headings-max-depth  50)
+        (files (list (current-buffer))))
+    (helm :sources (helm-source-org-headings-for-files files t)
           :candidate-number-limit 99999
           :truncate-lines helm-org-truncate-lines
           :buffer "*helm org parent headings*")))

@thierryvolpiatto
Copy link
Member

helm-org-startup-visibility have been disabled in this patch, to re-enable it it would need some more work. Also if you set completion-styles to flex or helm-flex depending you use respectively emacs-27 or emacs-26 you have flex matching (aka fuzzy) for free.
@alphapapa ping.

@thierryvolpiatto
Copy link
Member

I started a PR at emacs-helm/helm-org#10

@thierryvolpiatto
Copy link
Member

@matthuszagh have a look at emacs-helm/helm-org#10
and let me know if it is better with your "huge org files".
Thanks.
PS: Just get helm-org.el from there and load the file.

@alphapapa
Copy link
Member

alphapapa commented Nov 29, 2019

@matthuszagh I recommend trying the helm-org-ql command from the org-ql package.
https://github.com/alphapapa/org-ql I use it to search my collection of huge Org files, and it's very fast due to its built-in search predicate optimizations and caching.

@matthuszagh
Copy link

@alphapapa thanks for the recommendation! I'll try it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants